[any language]Is it possible to crawl a website and download targeting data on excel?

Discussion in 'Microsoft Office' started by durchnacht, Aug 4, 2016.

  1. durchnacht

    durchnacht MDL Novice

    Jul 12, 2016
    7
    3
    0
    Hi,

    I'm looking for the code script that I can use to crawl up specific website board and download targeting data.



    I'm a univ student and recently doing a research work on my major: economics.

    The goal of the research is to estimate the scale of underground economies of mobile phone subsidy, which is a very unique phenomenon in Korea's mobile phone market.

    It is a common practice where the giant phone makers like Samsung and telecommunication companies like KT subsidize people who have BETTER information about the REAL price of mobile phone with illegal subsidy.

    Only few who know the stuff can get these subsidy, resulting in buying new cell phones like Galaxy S7 for only about $300.


    Anyway, I'm trying to reach this goal by crawling up the price data people secretly share on board of the specific online website.

    To do this, I have to first crawl up all the posts with subsidy-related slangs in the title or the comments and then download the targeting words or price written in the post.


    However, I'm not a coder and cannot write the code without any help.

    So I'm wondering if there is any similar code script with similar intention that I can utilize.

    or at least any source where I can get some help.




    ----

    I'm a newb in Coding Life forum since I've never left Computing Life forum until now.

    Seems nice!
     
  2. Michaela Joy

    Michaela Joy MDL Crazy Lady

    Jul 26, 2012
    3,505
    3,688
    120
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. ofernandofilo

    ofernandofilo MDL Member

    Sep 26, 2015
    211
    128
    10
    #3 ofernandofilo, Aug 5, 2016
    Last edited: Aug 5, 2016
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. durchnacht

    durchnacht MDL Novice

    Jul 12, 2016
    7
    3
    0
    My knowledge base couldn't possibly afford coding a well working crawler for my project so I asked my good friend to do it for me in exchange for 3 free dinner. The result was satisfying but I'd like to do it on my own in the future so I think what you've listed can really help before I get to make a program that actually works. ;)

    I really like MDL community cause people are so nice unlike the other. Thx for your comment!
     
  5. durchnacht

    durchnacht MDL Novice

    Jul 12, 2016
    7
    3
    0
    i left a thank you reply almost instantly with my mobile but i don't know why that didn't work. I tried to make my own crawler w/ java referring your first link but my elementary understanding on the language kept me from doing so. :(
     
  6. vyvojar

    vyvojar MDL Novice

    Aug 10, 2016
    23
    25
    0
    You're looking for specialized crawlers, mere mirror tools are useless. Forget about Java too, it is horrible for this sort of stuff.

    A very basic scraper: hxxp://docs.python-guide.org/en/latest/scenarios/scrape/
    More powerful scraping framework: hxxp://scrapy.org/

    In case you're trying to scrape sites which obfuscate the data via javascript, you need a browser-based scraper, such as this hxxp://casperjs.org/ - with those, you just deal with DOM directly after 'navigating' and 'clicking' stuff.

    (sorry for no clickable links, low post count).