Website domain crawling

I need to so some large scale website crawling for data mining purposes. It needs to scale to potentially millions of sites. Unless you have done a project like this, please do not bid. I need an expert that has done it before. Not looking for a custom solution. I would expect that solutions exist that can be leveraged. Specs: - Input is a list of URL's  - Each URL is FULLY crawled starting at the depth provided. Example if [login to view URL] is provided, only the data under [login to view URL] is crawled, it would not see [login to view URL] -All pages are saved as HTML in a network file system under the top level target domain directory -All pages and links are crawled, including pages that require a click via javascript or user interaction -All pages are saved as their rendered versions as HTML -Child pages that are behind a javascript link: When this happens, the links are converted to an HTML link and inserted into the page. The rendered child page is saved as HTML -Inserted HTML links (javascript click) should be human readable format for QA Another process will consume and process the directory tree after each site is done. The crucial piece is that everything is traversable by following a regular html link. Please suggest what technologies you would use to accomplish this project. Please show that you have read this fully by putting your favorite color in CAPS as the first word in your bid. Do not post generic list of projects. I want to know what you have done in site crawling specifically. Thanks