Create a Web Crawler/Scraper for my online business - Experts Needed
min 50 $ USD / heure
I have been a web developer and digital marketer for the last 12 years and have been learning python for the last 8 months. The reason for this is that I am very interested in having a web scraper built for my web design / digital marketing company.
This web scraper will be very basic initially and I will need to understand the cost of running the scrape and fixing scraping issues before I can scale.
Here's what I need it to do:
• Crawl the internet looking for WordPress websites
• Add URL to dataset, listing whether the website is WordPress or not
• Filter those WordPress websites by page size add the page size number to the dataset
• Scrape the home page loading speed of website (using [login to view URL] or any other more cost effective alternative)
• Add the page loading speed time in seconds to the dataset
• Create a file for that data to live to be extracted and inputted into a table on my website for review. The table will include the URL, Technology (WordPress or not), Number of Page, Loading Speed.
To first understand if this is cost efficient, I will need help with the following questions
1. Which server is best to use for price and efficiency (current thinking in AWS)?
2. Is Python the best language to use for the scrape? (ruby, python etc)
3. What monthly cost can I expect? (please give a variety of pricing points based on output)
4. How many websites/web pages would I expect to scrape and analyse each month for these different pricing points?
5. How much would the storage space cost for these different pricing points?
6. How much would it cost to build my own website loading speed checker and is this cost efficient and reliable?
7. What would we need to do to speed up the crawl and be more efficient?
8. How can I be a good citizen on the internet and what safeguards do I need to put in place?
9. What is the difference in price if we add additional elements to the scrape (please see below)
I will be looking to run a test script using scrapy on my computer first to understand the kind of quality of websites I can expect to receive on the first 500 websites and would need help setting that up. This would not include creating the website functionality just yet. If the results are good, I will add other information to the scripts and also look to move this onto a server and build out the website.
To be suitable for this role, you must be able to answer these questions and have at least 3 years experience in web scraping, using API and building similar web crawlers.
Skills required
• Python
• Scrapy
• Beautiful Soup
• Selenium
• Panda
• Experience working with API’s
• Experience working with data analyse (Numpy and Panda)
List of additional elements we can scrape
• Use of themes in the design (find themes/ in content)
• Number of social shares
• Whether there is a video on the website
• Web Technologies used other that WordPress
• Marketing Technologies used like Google Analytics
Time Frame for initial delivery
• 2 weeks
If you are interested in discussing this further then please contact me with a rough guesstimate to set up the initial phase on my computer.
Thank you in advance.
Nº du projet : #26249625
À propos du projet
19 freelances font une offre moyenne de 52 $/heure pour ce travail
Hi This is vasilatos. Basically I understood your concerns, and I am very interested in your job descriptions. I think that I am fit for your job description compatible with your remarkable business. As a senior full Plus
Hello There, Greetings. We have experience in developing a web crawler that was able to extract data from 4 layers of the website. Dragon SRCH is a digital platform designed to facilitate technology data research. Our Plus
Hello! Sir Very interesting in your project. My name is konstiatyn who is professional in webscrapping. I have read your description and understand your idea I am specialized in webscrapping. I did so many scrapping w Plus
Hello Sir , My name is Vishal and i can help you with webscarping . I am expert at webscraping and i can edtract data from any site and put it in Various form like csv,excel,json For demo work you can message me. I wi Plus
Hi, I have gone through all your project requirements, it jumped out at me. I have 3+ years experience working with python. I recently completed a django project on which i integrated AWS. Python is actually the best Plus
Hi, Thanks for sharing your requirements here. I’m an Experienced Freelancer with a demonstrated history of working in the internet industry. Skilled in .Net, C#, Power BI, Angular, VueJs, React JS, Hadoop, Core PHP, Plus
