Atrax Mercator web spider crawlor clone

web spider crawlor clone Hello, I am looking for serious high performance search engin spider crawlorexpert software engineer. I am looking for a software engineer who has already worked and has experience in this kind of high performance spider cralor only. This means I am looking for a qualified expert only with refferences in the crawling industry!!! I need a crawlor clone of Atrax software and suplemented why not with Atrax software: extension to Mercator and Mercator software that - combines several Mercators - URL hashing, and off-line URL chec Atrax is a distributed crawler written in C++ and Python, which is composed of a "crawl manager", one or more "downloaders" and one or more "DNS resolvers". Collected URLs are added to a queue on disk, and processed later to search for seen URLs in batch mode. The politeness policy considers both third and second level domains (e.g.: [login to view URL] and [login to view URL] are third level domains) because third level domains are usually hosted by the same Web server. Tools/languages for implementation: • Scripting languages (Python, Perl) • Java (performance tuning tricky) • C/C++ with sockets (low-level)