Web Crawler/Scapping tool
par
luisurraca
A web tool that crawled/scrapped a list of URL (client asked 200k, but does a lot more than that) and saved some data in the database depending on some keywords found on the websites. Parallel processing was a must since 200k sites would take a lot in serial processing, with 25 concurrent jobs was able to get 20k sites scrapped in under an hour that could even higher depending in server resources.
Me concernant
Software Developer, specialized in web development (Ruby on Rails, Wordpress, Cross-platform HTML5 Mobile Apps)
