Icecat ([url removed, login to view]) provides online shops and price comparison sites with structured product data for computers, electronic consumer goods, cameras, mobile phones and other gadgets. I need somebody who uses Icecats XML interface to store product data from selected product categories in a mySQL database (that has to be build up) and makes this data accessible for further processes. A presentation of Icecat and its XML interface can be found here: [url removed, login to view]
This is the job:
Download/transfer data from all products of these product categories:
- Notebooks (Category ID 151)
- Netbooks (Category ID 1561)
- Mobile & Smartphones (Category ID 119)
- Digital Cameras (Category ID 575)
- LCD TV (Category ID 1078)
- Plasma TV (Category ID 903)
- GPS navigation systems (Category ID 910)
Please notice that:
- I only need the product categories in the German language version (LanguageID= „4“)
- Only structured product data are needed (Product Quality = „ICECAT“)
- Product data of related products („ProductRelated“) are not needed neither
- To start with, the database will have to be build and tested with the free available brands („Open Icecat“). At the final project stage we would switch to the entire Icecat data base that includes all brands („Full Icecat“). The data structure is identical, only the XML addresses have to be changed.
Automatic update of the mySQL database
Icecat provides an update XML on a daily basis (*[url removed, login to view]) that contains new and obsolete products.
With respect to daily updates you will have to write a routine that
- updates the mySQL database on a daily basis for the seven product categories mentioned above (add new, remove old products),
- updates are only needed for the German market (<Country Market value = "DE">)
- sends a notification mail after each daily update with a .csv file attached stating which products have been added and which products have been removed (short information for each product such as date, category, added/removed, brand, product name, EAN, Icecat product ID)
Browser based Admin view to enable non IT experts to handle the mySQL database
This admin view should enable the following actions:
- export of product data sets in a .csv data file with filters to select
o which product data sets to export (e.g. category, date)
o which product features to include in the export (all, brand, specific product features etc.). For each product category this selection can be stored as a default for future exports.
- Direct search for specific products (e.g. brand, product name, EAN, Icecat product ID)
- Manual upload of additional products (within the given categories) with .csv data files structured according to the product information in the category
- Note: given a solid functionality of this admin view and possibilities to select and export data sets filtering for date ranges, the “mail notification functionality after uploads” mentioned above could be omitted.
Support with hosting and configuration
You should help me to choose the right selection and evaluation criteria for hosting providers and their services. Furthermore, I will need support to install and configure the database with a German provider.
You will have to transfer all rights for the developed services to me.
In case of a successful cooperation there will be more programming to be done for me (integration of Amazon API and price comparison services, other language versions, personal account services and many more).
You should state a fix price and a termination date for this project in your bid. Please mention references to past projects related to this one.
A sound knowledge of Perl is preferred but not mandatory as my services will use Perl to work with the Icecat product data.
Please take notice of the following additional remarks and confirm your bid, if you wish to go ahead:
- According to some feedback I got, you have to handle very high data volumes when extracting data from Icecat. Please make sure you have the appropriate infrastructure to do this. Make sure you have fully understood the way data can be extracted from Icecat. In case of doubt, please check the attached manual for this.
- The MySQL database to be build by you will serve as the data basis of a web based product recommendation service and has to be able to handle a large number of requests and thus has to perform very well. Each user will cause calculation of (max # of products, e.g. 2,000 - 10,000) x (15 most important datafields) to arrive to his results.
- You will have to provide proper documentation of the MySQL database, its admin view and the code used for extracting and updating data from Icecat so that other web designers will be able to build on your work.
Specification on the browser based admin view of the MySQL database:
- To select data files, up to five data fields to be determined by the user can be selected and combined with “and” statements. Each of this five selection fields will enable search functions with “=”, “<”, “>” with respect to the given search value.
- Search results can either be listed on screen or exported as .csv file
Terms of payment:
The project consists of three stages so payment is made in three installments:
1) 30%: After MySQL database and its admin view are completed and fully functional. Proof of work completed: Database is setup and contains and least 5 products in each of the mentioned product categories, browser based admin view is working. At this project stage the database can be hosted at any location of the bidder’s choice.
2) Next 40%: After integration of all Free Icecat data, database is transferred to a public hosting service in Germany. Proof of work completed: All data from Free Icecat are in the database and can be accessed through the admin view; daily updates are running smoothly, operations will run without error for seven consecutive days.
3) Last 30%: After integration of all Full Icecat data and project documentation. Proof of work completed: All data from Full Icecat are in the database and can be accessed through the admin view; daily updates are running smoothly, operations will run without error for seven consecutive days, documentation completed.
For each project stage there is a max number of 28 days allowed (I would appreciate faster completion). If the bidder fails to accomplish the tasks within this period (as stated in proof of work) the work will be considered as not completed and no payment will be made. If a project stage is not completed the bidder may not proceed to the next stage. Milestone payments are made according to the outstanding part of each project stage.
Please also provide me with short information on how and at what costs you can support me with ongoing maintenance work, for example when Icecat changes its data structure.