A simple web crawler
- Collects the links from the websites
- Save the requested information in a mySQL database
- It must be possible to run mutiple crawlers
- The program must use threads for speed!
- Respect the robots exclusion standard
- Crawling depth
- Number of threads
- User agent of the crawler
- List of domains that will be crawled
Information to collect:
- Website Title:
- Meta Description:
- Meta Keywords:
15 freelance font une offre moyenne de $206 pour ce travail
Hi, I have done a web site which is very similar to what you are asking for. See PM for more details.
Custom software development - The Administrator removed this message due to advertising which breaches our Terms and Conditions.
I have confident for this project as recently I make a Plagiarism Software that heavily need the web crawler.
I have been creating such bots from a long time now. I assure you quality work and 100% satisfaction guaranteed. I've read your requirements and easily write such a crawler in java. Please check your PMB.
"SCRAPER AND CRAWLER HERE". We are new to freelancer.com so don't have reviews here. But you can check our reviews in elance, odesk and scriptlance. Please check your PMB for details.