Website Scraper and Import Tool

Dear Coders,

IMPORTANT: Please do not use placeholder bids and respond only with specific questions about my project. If your bid exceeds our limit, then do not bid.

We have a csv file containing several thousand tool model numbers. We want you to build a scraper that we can periodically use to scrape the individual URL for each product from 14 retailers and append the csv file with the URL's for each product.

If the product is not available on one of the source sites after attempting various combinations for the product model, then simply skip.

The 14 sites will be fixed but we'll need to be able to omit one or more of the retailer sites from collecting the data if necessary...a simple checkbox option next to the 14 retailers is all we'll require. That default they are all checked and hence all scraped for the product URL however if one or more is unchecked, they will be excluded from the scraping process and hence results.

You'll need to factor in how to prevent the scraper from being blocked. For this, we'll provide a set of proxies - we'll need to be able to either add a list proxies or add a URL for rotating proxies. I would also like you to build some functionality that reports or logs when proxies are failing. Preferably, you should be able to offer advice as to the most suitable proxy provider. Most importantly, do not build for one type of proxy but give options.

It is important to note that some of the supplier sites below use CloudFlare making it difficult to scrape. You'll need to find a way around this.

We've included a sample csv file with product examples. So far for testing purposes, I have provided a list of about 60 items. In the final csv, there will be many thousands of products to scrape. Your scraper must run fast.

The structure of the input csv is as follows:

Product Title, Model Number, Brand, Category, Sub-Category, Product Image

The 14 retailers are listed here:

1) Total Tools - [login to view URL]

2) Just Tools - [login to view URL]

3) Vek Tools - [login to view URL]

4) Get Tools Direct - [login to view URL]

5) Power Tools Specialist - [login to view URL]

6) Bunnings - [login to view URL]

7) Discount Trader - [login to view URL]

8) Tool Mart - [login to view URL]

9) Melbourne Tools - [login to view URL]

10) Trade Tools - [login to view URL]

11) Australian Industrial Supplies - [login to view URL]

12) C L Tool Centre - [login to view URL]

13) Gasweld - [login to view URL]

14) Mitre 10 - [login to view URL]

My suggestion is to use the model Number in the search bar of the retailers below to find a suitable match for the product. If you have a better solution please advise.

Please be aware that should the first search for the item fail, you may need to use an alternate product number combination. For example, if you can't find D26500K-XE, then you should try D26500K(XE) or D26500KXE or something similar. It will be your job to determine the best way to find the products taking into consideration the unique way each site represents it's products.

Next, you'll need to modify a price comparison plugin we have built to allow it to import this csv data into the database. We will provide access to a copy of the site for you to review the code after acceptance of the project.

Time Frame: 2 weeks absolute maximum.

Any questions, please advise.


Compétences : HTML, Javascript, PHP, Architecture Logicielle, Web Scraping

en voir plus : web scraper multiple pages, mac web scraper, free, web scraper tool, download, web scraping software, scrape images from website, web scraper chrome, website automated test tool accessibility, best website scraper, web based excel import tool, joomla import tool, website import tool joomla, import tool magento import contents oscommerce, magento import tool, wiktionary import tool, simple website scraper, website baker import html files, ebay import tool zen cart, import tool joomla article

Concernant l'employeur :
( 336 commentaires ) Diamond Creek, Australia

Nº du projet : #17537050

39 freelance font une offre moyenne de $618 pour ce travail


Hi, i have done a script to scrape 7 websitez from your [login to view URL] check my profile portfolio and you'll an excel with green/red boxes. I will recommend good proxy provider foe that that has auto rotating ip. Also re Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 15 jours
(256 Commentaires)

Dear sir, This is Lin, i did more than 380+ scraping project, please view my profile page here then you will know. Can we discuss more details about your project? t Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 6 jours
(386 Commentaires)

Hi I understand your requirements and we can utilize proxies to avoid the blocking. I bid each site with 125 USD Can work on a demo before you award me. Can finish it in 5-7 days Thanks

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(203 Commentaires)

Hello Simon, This is Vaishali from "Hire WordPress Experts" and I am here to help you with Website Scraper and Import Tool . We have gone through the information provided by you. and I can assure you that we are ex Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(282 Commentaires)

Hello, Dear How are you? I am very interested in your project and am ready for starting your project for now. I have experienced in developing HTML, Javascript, PHP, Software Architecture, Web Scraping. I will work Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(77 Commentaires)

Hi I am very,very interested in your project My name is Shan Bin and I'm a Chinese developer. I have 6 years of web scraper development experience such this projects. And I have good skills with this program languag Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(111 Commentaires)

Dear Sir, I'm very much delighted to let you know that i did data scraping with PHP-cURL, PhantomJS, Node.js, Selenium from many sites. I just scraped the data from web site and then wrote the data in mysql database Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 15 jours
(84 Commentaires)

--^^-- Hello! ***************************** Nice to meet you! ***************************** I am a Web Scrap Professional developer. Scrapping and importing tools. I can meet your requirements fully in a short Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(107 Commentaires)

Hi, I can start work on your project immediately once you confirm me and i am also open for further discussion if you want. I have 8+ years in web design & development with good experience in following skills ... Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(127 Commentaires)

Dear scm1971, I do have relative experience achieving something like yours "Website Scraper and Import Tool" thusly I am expecting discuss and progress. Relevant Skills and Experience I have 10+ years of experi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 12 jours
(44 Commentaires)

Hello sir, please reply soon , i will start work right now and complete all soon, please share website details, thanks

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 0 jours
(213 Commentaires)

Hello, I can help with you in your project Website Scraper and Import Tool I have more than 5 years of experience in HTML, Javascript, PHP, Software Architecture, Web Scraping. We have worked on several similar p Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(57 Commentaires)

Hi Sir, I’m a Python Programmer. I have created python script for bunnings and totaltools for searching product sku's. as you know every website has unique structure. hence we used to analyse everywebsite paths and co Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(105 Commentaires)

Hello, I can scrape all the 14 retailer websites and get the date for you CSV file. As you said you will provide a set of proxies, I can rotate the proxies and use it with those websites which blocks us. There are se Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(74 Commentaires)

Hello. My name is Maciej. I am a Python developer and web scraping expert. I can create the scraper matching requirements for the price bidded. Please contact me if you are interested. Kind Regards Maciej J

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 20 jours
(62 Commentaires)

*** I checked your description, website and attach file. I can build scraper tool. I want to discuss with you about this project. thanks.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(31 Commentaires)

[ Just Relax and I Do All What You Need } This link is wordpress site we created [login to view URL] Please check this as a sample You can visit that link as a sample of what I did Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(36 Commentaires)

Hello Sir, I have read your job description regarding Website Scraper and Import Tool. I have 6+ year of experience in HTML, Javascript, PHP, Software Architecture, Web Scraping. I have strong expertise to accomp Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(99 Commentaires)

Hi, I'm a full stack Web Designer and Developer. I am professional in Web Designing, having 12+ years experienced in Website Scraping and Importing Tools. I have see your attachment file, reference URL and expert Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 8 jours
(32 Commentaires)

Hello There, i go through with your all requirement and understand it very well, i make sure i will give you best result, I've experience with scrap websites, i will also build some functionality for reports and Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 15 jours
(19 Commentaires)