Fermé

Developing a file scraper to scrape firmware files from vendors websites

Project:

The project is a scraper which is able to continuously crawl different vendors websites and find various kinds of IoT/connected devices firmware files available on vendors web sites and download the files as well as their details in a centralized database. It should be able to download only the binary firmware files not text files such as release notes or firmware installers, be able to download new files as soon as they are available on the vendors web site( the time to send a new request to sync should be able to be configured per vendor), and be able to support adding credentials or the other required info by the vendor to be able to download the firmware file or get its details. In case a vendor provides an installer for the specific firmware, and not the firmware itself, if the download link can be extracted from the installer, it should be found out and the firmware should be downloaded. If there is no way to receive the firmware without installation, the installer should be simply downloaded and marked as “installer” in the database ( which needs another binary field called "installer" to be added to the database design ) It should also be clear in the DB that for which vendor & firmware this installer is. Note that:

1. The code should be written in Python

2. The system which the script needs to be executed does not have any GUI, so it should not need installing any GUI app as a dependency.

3. The list of vendors will be provided by us and it must be able to download all of the firmwares for all of the devices as well as all of the different versions of a specific firmware available on the vendors websites without ignoring any of them. We are open to any suggestions provided by the Developer regarding new vendors but developing the scraper for them requires confirmation from us.

Output:

The output generated by the system should be able to download the firmware files in the path we define and save their details in a SQLite database. The mandatory database fields include ( Manufacturer, Model, Version, Type, Name, Release Date(if available) ) i.e. ( Cisco, Video Surveillance 6030 IP Camera, 2.7.0, IP Camera, [login to view URL], 21/08/2015 ) There is a non-mandatory binary field which indicates if the device is discontinued or not depending on the fact that vendor mention that on the website or not.

Deliverable:

1) Python Source Code including the comments in the code explaining each function & its details. We should be able to give any required input as an argument and execute it as one line command in the Linux terminal.

2) Dependencies

3) Manual to install, configure and use the scraper

Compétences : Python, SQLite, Web Scraping, Linux

Concernant l'employeur :
( 0 commentaires ) Schaarbeek, Belgium

Nº du projet : #21370468

54 freelance font une offre moyenne de €1087 pour ce travail

chirgeo

Hi. I did read the project description and have a few questions. 1. Do you need the script as well or data only? 2. What is the format of the output data? CSV is OK? We can do other formats as well. 3. Which fields do Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(115 Commentaires)
7.4
bestit4u

Hello Respect sir I saw your description, I can't stand the excitement of seeing the project you post on... I bet one's shirt on I have the ability to complete your project perfetly because I have experience with your Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(90 Commentaires)
6.9
DevStar925

Hi, I'm very interested in your project as I have rich experience in web scrapping for such features based on Python language. I would like to discuss more details via chat. Thanks

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(29 Commentaires)
6.6
adeelpirzada

Hi, I hope you're doing well i have done scrapping almost on Half of Worldwide web including eCommerce giants (Amazon, eBay, craigslist) News Feed, Social media websites, API's. I develop my own scrapers and Bo Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(35 Commentaires)
6.5
susanna2018

Hi, Sir!! i am a python expert and full-stack developer with full time. -):-):-) i want to help you with my python skills :-):-):-) “””” Specially Data Science , Data Analysis “””” i use sklearn , Tensorflow, k Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(47 Commentaires)
6.0
Mickelson

Dear my employer! I have just read your description carefully. As a python expert(Python Level 1: 75%, https://www.freelancer.com/u/Mickelson?w=f ), I am sure that I can handle your project perfectly. Even If you don't Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(73 Commentaires)
6.4
Nidhijain123

Hello, I have gone through your project description and understood what has to be done in this task and can give you a complete assurance that I can do the job. I have sound experience in performing such kind of work Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(21 Commentaires)
5.5
zeke

I have worked on many similar projects developing Python scripts to extract all kinds of information from all kinds of web sites including those with complex and obfuscated javascript. I am absolutely confident than I Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(41 Commentaires)
5.8
RobustTechServ

Hi there! My name is Orestes and I am a Software Engineer with specialty in Java (Spring Framework) and PHP (Laravel Framework). I represent a whole team of software engineers (so our language stack is actually quite Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 20 jours
(11 Commentaires)
5.6
smsaurabhv

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 3 jours
(62 Commentaires)
5.3
JoBergs

Hello, i have knowledge in all areas required to build this application for you. I know Python well, i have build scrapers with download functionality with Scrapy , i'm using Linux for many years and i know some softw Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 10 jours
(24 Commentaires)
5.2
AlexanderPGR

hi, there. I have read your description carefully. I am very interested in your project. I have high Python skill and have experience with web scraping. Please contact me and discuss further. Thanks.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(20 Commentaires)
5.2
shaktistanwar

Hi I am a technical evangelist, developer and can do this task for you. I have more than 15 years of experience in software development and am a microsoft certified professional.I have strong work experience in xamarin Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(7 Commentaires)
5.4
ganeshrasekar

I can do it

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(12 Commentaires)
4.3
ThilinaG01

Hi, I build a python scraper to scrape vendors websites for firmware files. please send me a message so that we can discuss [login to view URL],

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 21 jours
(14 Commentaires)
4.4
arhamsoftltd

Hi, I can design and develop a scrapper for you in Python I am familiar with related libraries and frameworks such as Tensorflow, Keras, Theano, Torch Scipy, Scikit-learn. and I am skilled in Python, R, Lua, Go etc. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 35 jours
(3 Commentaires)
4.5
MaMingMing

Hi, respected client !! This is Ma, energetic full stack developer with 7+ years experience developing robust code for high-volume businesses. I am a relentless learner, and enjoy reading about, studying, and attending Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(3 Commentaires)
4.5
Guptapuru304

Hi! I'm senior python developer, working for 3 years now. I am expert in python automation using selenium for chrome and web scraping which I have done for sites like amazon, facebook, ebay etc. I can complete the pr Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(13 Commentaires)
4.5
coolmob0816

Hello, I am a senior web developer with rich experiences about web scraping and data mining and handling. I have read your project detail carefully. Before I have scraped data according to employer's request with pytho Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 15 jours
(6 Commentaires)
4.0
topexpert713

Hi, Nice to meet you! I am very interesting your project and I am confident of I can help your job. I'm a professional python expert developer with over 7 years of experience. Seems to be an interesting work and good Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% EUR en 7 jours
(10 Commentaires)
4.0