website scraper, spider and downloader

We require a program that can download files and web pages from a list of web sites.


We have a list of 65,000 websites

In most of these web site, they contain files, mostly in PDF format, that we need to download

These files or web pages will contain a specific word or phrase, either in the URL or the web page.

The web pages may have many different file extensions, such as .asp, .aspx, .php, .htm, etc


1. We need to have the ability to load in a list of websites to be searched

2. We need to have the ability to set the number of levels the spider will look into the website

3. We need to have the ability to determine one or more file types

4. We need to have the ability to download files that match a specific criteria, such as word or phrase in the url

5. We need to have the option of creating and downloading files into a folder with the website address name

6. We need to have the option of staying within the base website, or 1, 2, 3, etc website address away from the base url

7. We need to have a time out option, to stop the spidering/downloading after xx seconds of not finding any files, and moving to the next website on the list.

Look at the download/spider program called Medusa, by Candego.com. Many of the options on this program are required on our program.

This program must be able to operate on current Windows PC and NOT be dependent on a web browser.

Program must be completed within two weeks of project award.

Compétences : Exploitation de Données, Web Scraping

en voir plus : program download websites, program download website, finding match, creating website url, asp spider, web browser download, list of website creating website, spider, scraping pdf, finding a pdf, browser current url, data scraping word, address name format, website scraping php, downloader data, data mining scraping website, number scraper, program downloader, program pdf scraper, name scraping website, web spider project, download project web browser, scraper site data, download options data, url spider

Concernant l'employeur :
( 76 commentaires ) Edina, United States

Nº du projet : #2388974

12 freelance font une offre moyenne de $579 pour ce travail


I look forward to discuss further

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 22 jours
(63 Commentaires)

I specialize in similar projects. Please check PM for more details.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(52 Commentaires)

I am confident to handle your project, please check your inbox for details. Thank you.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(8 Commentaires)

Will write a bot which will crawl your site list, just need the site list to crawl them. Thanks.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(3 Commentaires)

Hi! it will be pleasure for me to this work for you :)

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(1 Évaluation)

Hey, I can write you a crawler in C++/Qt. It'll be cross-platform and fully browser independent.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(1 Évaluation)

Dear Sir, I can make a spider which exactly suits your needs. for more info please check your private message box. Thanks and regards

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 15 jours
(1 Évaluation)

I will satisfy your needs. The script will be exactly as you want. I am an expert in web scraping, so it will be really easy for me.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(0 Commentaires)

Please check the PMB

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(0 Commentaires)

Can be done.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 30 jours
(0 Commentaires)

Please check the PMB

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(0 Commentaires)

Hi, I have a code that does what you're asking in the project. Details on PM Regards

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(0 Commentaires)