Real-time web scraping multiple websites

I need a solution for extraction data in "near real time" from different websites but with the same general topic.

I don't know the numbers of starting pages (maybe 500.000)

The software should do:

- Autodiscovery new URLs (based on specific rules)

- Extract and save some particular data (object name, description, price, ...) and general data (HTML title, author, ...)

- These data must be updated continuously

- These data should be saved in a database

- I need a JSON API that should inquiry that database with these functionalities

- parametric search

- aggregate search

- full-text search

- In case of error or new needs, It should be reprocessed all or some pages.

- Easy to add new sites.

- Enable to scrape ajax based websites.

I'd like to have a solution that can be easy to install, configure and most of all easy to update in case of errors.

Compétences : AJAX, JSON, Web Scraping

en voir plus : web scraper chrome, how to scrape data from a website, real time web scraping python, web scraping api python, web scraping applications, visual web scraper, web scraping api free, web scraping tutorial

Concernant l'employeur :
( 2 commentaires ) varese, Italy

Nº du projet : #19710624

57 freelance font une offre moyenne de €3756 pour ce travail


Hi, Can you share more details about your project? I think Python, exactly scrapy is the best for your project. As a professional full stack developer who has rich experience of Web scraping, I'm very interested in Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(29 Commentaires)

Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields do Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 5 jours
(39 Commentaires)

Hello? Nice to meet you. I am excited to work with you on this project. I am ready to start work immediately. Here is on of my scrap with python working result. [login to view URL] I can do python with seleniu Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(44 Commentaires)

Greetings. I have extensive knowledge and expertise in web scrapping/ crawling software. Would you please come over chat so that we can discuss this project further? Willing to showcase my work over chat. It seems like Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(11 Commentaires)

Hi, I am Fatima a full time PREFERRED FREELANCER and Python certified developer with 100% guaranteed delivery. Before we start the project would like to have a detailed conversation with you regarding this project to Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(88 Commentaires)

Hello there! I would like to be considered for your project. We can build a script to scrape multiple websites according to your requirement.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 8 jours
(95 Commentaires)

Hello! Collecting data from various places is my profession. I am expert in Data extraction, validation, mapping, Data Analysis & reporting scrapping using C# and Python. I develop my own scrapers and Bots. I have expe Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(27 Commentaires)

Greetings, I am an experienced professional scrapper and have done similar projects in the past. Same can be verified from my profile. Let me allow to assist you with your requirements. Thanks

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 8 jours
(45 Commentaires)

Hi I am a technical evangelist, developer and can do this task for you. I have more than 12 years of experience in software development and am a Microsoft certified professional.I have strong work experience in Nopcom Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(4 Commentaires)

Hi there, I have read initial brief & wondering if you can please provide me URL of those websites, from where you want to scrap the data? Please click on the chat button for further discussion about your project Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 70 jours
(5 Commentaires)

Hi there how are you I am a senior python developer and i have used python and selenium . Please contact me and tell me about your scraping project . thanks for your posting

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 2 jours
(19 Commentaires)

Hello, and thanks for the opportunity to bid on your project. Please check my portfolio and reviews from other customers https://www.freelancer.com/u/TenStar718.html I am an expert in many different area’s of web Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(10 Commentaires)

Hello, I am Python expert and expereienced on scraping web sites. I can build a nice app for your usage. Can we discuss details over chat please? Best Regards

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(37 Commentaires)

** WE ARE SCRAPING SPECIALIZED ** Dear Client, I would like to inform you that We have already worked on similar kind of requirements as yours and We are confident that our skills and experience will definitely help Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(5 Commentaires)

Greatings, I would love to have the opportunity to discuss your project. Web scrapping is based on RegExp tech. I have full experience with Scrapping... real-time. I would like to work with you. Please message so Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(3 Commentaires)

Hello, Dear How are you? I have read your project description and am ready for discussing with you about project. I have experienced in developing AJAX, JSON, Web Scraping. I will work very hard and best for you. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(11 Commentaires)

Hi, I can design and develop a solution for data scrapping of differetn websites let's talk to decide further It will have a catchy and interactive landing page My 9 years of development experience in SAAS an Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 45 jours
(1 Évaluation)

Hi, As per understand your project requirement we found that you are looking real time data extraction for multiple website in same time. You want three type of search parametric search, aggregate search, full-text s Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(3 Commentaires)

Nice to meet you I have more than 15 years development experience and a great team, we can complete your project with clean code and best practices To be able to give clients what exactly they want, we are workin Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 5 jours
(11 Commentaires)

Hello, I have gone through your Project requirements and Yes I can do it and also assure you that I can do this job perfectly. Hope to hear you soon. Thank you.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 30 jours
(8 Commentaires)