This needs to be completed using Scrapy and Python 3. Please do not bid on this project if you have never used Scrapy and Python 3 on your development machine.
I need you to build crawlers for the websites [url removed, login to view] and recruitireland.com. Please visit both of these websites before bidding on this job.
* The crawlers should navigate though each job category (accounting, construction, etc.)
* The pagination should be used to navigate through each page of jobs.
* When the crawling is finished, you should save the results to a sqlite3 database, and send a status e-mail containing the number of jobs found in each category.
* Note when saving each job to the database, you should check if the job already exists in the database (probably by using its URL). If it already exists, you can skip saving that job.
* Similar functionality should be imported (e.g. use DRY programming).
* It might make sense to insert the jobs into the database in batches (e.g. each page of jobs) instead of one by one.
Any questions just ask.