Create a Script to Download All Listings Data from eBay Motors API in Python 3

We need to get all vehicle listings on eBay Motors via the eBay Motors API. The following fields need to be collected for each posted vehicle:

Year, make, model, trim, mileage, VIN, title type, body type, exterior color, transmission, number of cylinders, fuel type, number of doors, drive type, number of views, number of watchers, ad number, price, location, date posted, date updated, listing title, url, and description (text only).

We already have an API key.

Needs to be written in Python 3. Needs to run inside a Docker container.

We need the script. The script needs to be able to be repeatedly run in a Docker container.

The script needs to be able to run continuously by itself, or to wait an interval (number of minutes) after it finishes before beginning again.

The script needs to be able to search by all of the search filters that the ebay API offers. They need to be optional (such that it will run without any filters).

The script needs to be able to accept all of the following as optional arguments:

- filters from eBay

- debug mode

- verbose mode

- interval (in minutes) to wait before restarting

It needs to feed the data into our existing MySQL 5.7 2nd generation InnoDB database on Google Cloud Platform with the following table structure (which will be done using the Google Cloud SQL Proxy):

Please see the attached file for the database table structure / DDL and some details about the fields.

The API method returns only 10,000 results at a time, in pages of 100 results each. The API limits us to 5000 calls / day.

The script will need to work with these constraints to get all new listings and all updated listings since it was last run every time.

The call to the API should filter the results to:

- location = US, Canada

- ListingType = AuctionWithBIN, Classified, and FixedPrice

The script should store all times in UTC.

Each time the script is run, it needs to get the data from all the listings that have been created or updated since the last time it ran (even if the last time it ran was days or weeks ago).

Each time the script finds a listing that already exists in the database, it needs to update all fields in the row in the database. It should also update the date_updated field each time.

All fields need to have commas removed from them before they enter the database.

Any time a field does not have a value or has 'not specified' as the value, it should be stored as NULL.

The location field should be stored as city state zip

The script needs to output everything to a log file for debugging.

The scrape should break gracefully on Ctrl + C.

Script needs to use "source = 'ebay'" for SQL statements since the table contains records from other sources.

Each time the script finishes running, it should output some status elements (such as how many items were created vs updated).

Please let me know which days you will be working on my project, and how many hours / day.

The code should be self documenting (good method names, variable names, structure, etc.).

All of your code needs to be well commented.

Your code needs to have good error handling.

Your code needs to follow good programming practices and standards.

We will manage all code and issues via GitHub. You will use the Projects feature of GitHub to show work you have not started, work in progress, and work completed.

I expect that you will complete it in the timeframe you quote. Otherwise, we will cancel the contract.

Please feel free to ask any questions you need!

Compétences : Python, Web Scraping, Data Scraping, Data Extraction, Web API

en voir plus : ebay completed listings data, create script harvest data, create script net retrieve data, free ebay clone script download, create macro download data yahoo finance, python script download historical data yahoo, script download google finance data python, create youtube script download, ebay motors api create listing, create script extract data, collect ebay completed listings data, ebay motors php script, ebay motors data mining, script download data internet, create script data entry, php data collection script download, perl script download data, website script download data

Concernant l'employeur :
( 3 commentaires ) Riverton, United States

Nº du projet : #21155929

36 freelance font une offre moyenne de $195 pour ce travail


Hi Paul , I am interested in your project related to build a scraper for ebay Motors API. Please send me a message so that we can discuss all the details. Thanks, Ramzi

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 12 jours
(419 Commentaires)

Hi there, This is Lin, i did more than 380+ scraping project, please view my profile page here https://www.freelancer.com/u/mantislin.html then you will know. Can we discuss more details about your project? then i ca Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(288 Commentaires)

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 4 jours
(35 Commentaires)

Hi Nice to meet you. I have enough experience in python script. Below the libraries are I used in past project. selenium, pandas, matplotlib, lxml, beautifulsoup, scipy, and other useful libraries. I have written some Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(66 Commentaires)

Hello. ~~~ i CAN DO IT IN 6 HOURS ~~~ Please contact me. I' d love hear from you. Thanks Best Regards.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(44 Commentaires)

⭐⭐⭐Dear sir!⭐⭐⭐ I am very interested in your project and I am exciting. I read your project details carefully and I though that I am the best fit developer for your project. I have rich experience about your project, s Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(14 Commentaires)

Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHON Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(60 Commentaires)

Dear Employer I have extensive expereince in scraping product data from Amazon I can assure you I can fulfill all your requirements as per task description. Should you be interested please let me know and I would be Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(63 Commentaires)

HI, there! I have read your description carefully. I have high Python skill and can scrape data from the website. I can agree with your suggestion that I have to work via Github. Please contact me and discuss more.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(15 Commentaires)

Dear sir I have read the requirements and completely understand the project.I will build the script in python 3 and it will get the results using filters and then save into database according to your [login to view URL] nu Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(16 Commentaires)

Hi, I am Huang. I have worked on open source python ERP customizatio like Odoo and ERPNext and have got wealth of experience in ebay, amazon, usps and fedex api integration in python3. I can help you get product listi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(5 Commentaires)

Hello Sir, I am professional developer with having more than 5 year's experience in this field. I read your project description and I am interested in it I need some more information related to your project Can we disc Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(9 Commentaires)

Hello. I am a senior Python developer and I have plenty of experience in Python development. I have developed the web scraping app using Python3. And I have developed the python program for the data transfer between tw Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(13 Commentaires)

Hello! I’ve read your job description & I am very much interested in working on your project. I have a good experience of the works. I am the right candidates for this task. I am a quick, reliable and efficient worker Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(27 Commentaires)

Hello, I'm [login to view URL] python BeautifulSoup,Selenium,Scrapy is very simple for me. I can give you good delivery on time with good partnership. Please don't waste your time to auto bidding bot. Please send message and d Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(8 Commentaires)

Hi, Hope you are doing well! I have checked your requirement. I can do this project as i am professional Python,SCraping Expert. It will be our great pleasure to work on your project. I'll complete the work as per req Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(6 Commentaires)

Hi sir, i am experienced in scraping, i did many similar jobs which you can see in my reviews, i have developed some codes already so the speed will be high for your job, would you please share the details?

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(4 Commentaires)

Hi! I am very happy to see your project detail. I think it is just my project. :) As you can see in my profile, (my review: https://www.freelancer.com/u/fivestar10?w=f) I have much experience with your job. please cont Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(2 Commentaires)

Hello. Do you want scrapping result as csv file? I have tons of experience with web scrapping result by using python. lets discuss in detail over chat. Best regards.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(5 Commentaires)

Hi. Please survey my reviews. I saw your description carefully and I am confident I can full fill your requirement. So I provide my skills in hopes of working with you. I am full-stack developer and I have 2+ years Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(4 Commentaires)