Web Scraping of news outlets using C++ into NoSQL databases

$2-8 USD / hour

Completed

Posted

over 10 years ago

$2-8 USD / hour

We are looking for a programmer to develop a c++ scraper for financial newsblogs. This should be reasonably commented, and run with parallel threads. The program should: Authenticate itself (if necessary) on the website Create a JSON object saving the contents of the article Some websites that will be scraped are: The Wall Street Journal -[login to view URL] Seeking Alpha - [login to view URL] The Motley Fool - [login to view URL] ..more websites are to come, so the script should have generic elements and be easily extensible The results will be in JSON structure, preferably inserted into a mongoDB instance (couchDB may also be used), or for testing purposes json files.

Project ID: 5138634

About the project

3 proposals

Remote project

Active 10 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

Awarded to:

@nitelfreelance

Hi. Why are you going to use C++ for such purpose? Usually this language is used in system level apps. Javascript, java, perl and pyhon commonly are used for web scraping. We have done many scraping projects using python and scrapy framework; please see [login to view URL] It is a mature tool for scraping. Also we can use PhantomJS and javascript to do scraping. Parallelism would be straightforward, depending on the bandwidth and nature of the target site though. Thanks

$12 USD in 15 days

4.9

(18 reviews)

5.3

3 freelancers are bidding on average $20 USD/hour for this job

@julianrath

I have a lot of skills regarding C++/Network programming. I also have done some multiprocessing pipelines in C++ using boost. Looking forward to head from you. Best Regards, Julian David Rath

$38 USD in 5 days