I am looking for a software application that can crawl a list of websites for expired domains (any domains with a status of 404). The crawler should crawl a list of inputted names via a source text file. The source text file would look like this, 1 domain name per line:
[login to view URL]
[login to view URL]
[login to view URL]
[login to view URL]
Once software grabs the source domains, it should fully crawl each site and report to a csv file any expired domains it finds. The format for the csv should be:
domain where found, URL where found, expired domain
An example line would be:
[login to view URL],[login to view URL],[login to view URL]
The crawler should use just one thread, and for the user agent, should just be chrome if that is possible.
I plan to do a phase 2 later where the crawler will check an API to see if the expired domains are available for purchase, and also to add threads, proxies, and other features. This will all be phase 2 and 3.
Right now, I just want phase 1 completed where it takes an input of "[login to view URL]" and then for domains it finds in 404 status, it should output "[login to view URL]"
This needs to be desktop based running on Windows 10 Pro. It can run in Powershell or Python, and other languages are fine as well. For continued development, it would be beneficial to have it in Python, but any language you prefer is fine.
Please include these with your bid:
1. Project Price in Full to do phase 1
2. Time to complete, since I am really needing this soon
3. What language you will be using
4. Any relevant experience you have with crawlers
29 freelance font une offre moyenne de $169 pour ce travail
Hi, I can make this crawler for you, but I actually already providing my clients with expired domains with backlinks found on sites of their choosing. also for the architecture that you've outlined you'll need lots of Plus
Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have created similar crawler in the past for checking expired domains . I can do it again for you ! Check out my profile and former clients feedba Plus
Hi Sir, I have created same crawler 2 years ago for some employer and a lot of crawlers for other types, I can complete in 2 days. Im a Python Programmer. I have created many Bots to scrape the websites like Google, Al Plus
Hello. I have read your project description carefully and bid here. I am a senior python developer and have expertise in webscraping. I can complete your project in a day. I will use [login to view URL] for GUI. I can pro Plus
Dear As an expert in web scraping, I developed 500+ spiders using python, scrapy, php,nodejs,c# and proxies. And I scraped data and download images, files for CEOs and output result as csv, json, xml(yellow, ebay, wi Plus
Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Plus
Hello How are you I have read your job description in detail . I am Xu It is a piece of cake i have done it before Please contact me and Let's discuss more thanks for your posting . hope to hear from you .
Hi, 1. Project Price in Full to do phase 1 Ans: 150 USD. 2. Time to complete, since I am really needing this soon Ans: 1-2 Days maximum. 3. What language you will be using Ans: Python. 4. Any relevant experience you ha Plus
Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO Plus
Dear Client Thank you for taking for your precious time to looking for my bid. I have fully read the job description and it is very clear to me. I am a senior web scraper who is good at python. I am using pytho Plus
Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide Plus
I checked your description. As I am a software and scrap expert , I have a wealth of experience and skills in these projects. I prefer python language and i will complete it within 1~2 days. You will get good results. Plus
Hi I am a technical evangelist, developer and can do this task for you. I have more than 15 years of experience in software development and am a microsoft certified professional.I have strong work experience in Nopcom Plus
Hi, I am interested in taking up your project. I have done 10+ projects in python software development,website development,desktop automation,website automation and web scraping,.I deliver fast,clean and quality work Plus
Hello how are you? My name is Ugur Catak. I am a software developer. About your project: Since we don't need to download entire page, we can request for the status code only to speed the bulk query cycle. 1. Project Plus
1. Project Price in Full to do phase 1 $200 2. Time to complete, since I am really needing this soon two days 3. What language you will be using python 4. Any relevant experience you have with crawl Plus
Hi, I will be using python language to do this project. I work fast, and I will probably finish it in few hours, but there are always some small complications, so it will be take me maximum of 2 days. I have been web s Plus
Hi, My name is Dhiraj and I am a professional web scrapper and data mining expert in python with more than 6 years of experience. I have expertise in scraping complex/dynamic websites using requests, BeautifulSou Plus
Hi 1. 150$ 2. one day 3. NodeJs 4. I'm a senior developer and have good experiences on web crawling and scraping development. Although I've just started using this website, I have a lot of experience with Selenium Webd Plus
Hi I've carefully read the description of your project. I am very interested in your project As I am a professional Crawler Developer, I am sure I can help you I'm ready to start to work on your project right now. Plus