Fermé

Expired Domain Web Crawler Needed

I am looking for a software application that can crawl a list of websites for expired domains (any domains with a status of 404). The crawler should crawl a list of inputted names via a source text file. The source text file would look like this, 1 domain name per line:

[login to view URL]

[login to view URL]

[login to view URL]

[login to view URL]

Once software grabs the source domains, it should fully crawl each site and report to a csv file any expired domains it finds. The format for the csv should be:

domain where found, URL where found, expired domain

An example line would be:

[login to view URL],[login to view URL],[login to view URL]

The crawler should use just one thread, and for the user agent, should just be chrome if that is possible.

I plan to do a phase 2 later where the crawler will check an API to see if the expired domains are available for purchase, and also to add threads, proxies, and other features. This will all be phase 2 and 3.

Right now, I just want phase 1 completed where it takes an input of "[login to view URL]" and then for domains it finds in 404 status, it should output "[login to view URL]"

This needs to be desktop based running on Windows 10 Pro. It can run in Powershell or Python, and other languages are fine as well. For continued development, it would be beneficial to have it in Python, but any language you prefer is fine.

Please include these with your bid:

1. Project Price in Full to do phase 1

2. Time to complete, since I am really needing this soon

3. What language you will be using

4. Any relevant experience you have with crawlers

Compétences : Web Scraping

en voir plus : java web crawler image capture, web bloggers needed, web crawler cpu usage, write web crawler, expired domain checker, web crawler names, web sites needed, web designers needed, web crawler system requirement, web crawler demo gathering information function, java spring web crawler, web page sql database input, good sites web projects needed, simple input file php script web crawler, php web crawler input, expired domain crawler

Concernant l'employeur :
( 65 commentaires ) Loveland, United States

Nº du projet : #20031872

29 freelance font une offre moyenne de $169 pour ce travail

e3d

Hi, I can make this crawler for you, but I actually already providing my clients with expired domains with backlinks found on sites of their choosing. also for the architecture that you've outlined you'll need lots of Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(238 Commentaires)
8.7
zekovicm

Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have created similar crawler in the past for checking expired domains . I can do it again for you ! Check out my profile and former clients feedba Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(91 Commentaires)
7.1
devi222

Hi Sir, I have created same crawler 2 years ago for some employer and a lot of crawlers for other types, I can complete in 2 days. Im a Python Programmer. I have created many Bots to scrape the websites like Google, Al Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(119 Commentaires)
6.8
Bluesky122

Hello. I have read your project description carefully and bid here. I am a senior python developer and have expertise in webscraping. I can complete your project in a day. I will use [login to view URL] for GUI. I can pro Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(29 Commentaires)
5.8
yongjin818

Dear As an expert in web scraping, I developed 500+ spiders using python, scrapy, php,nodejs,c# and proxies. And I scraped data and download images, files for CEOs and output result as csv, json, xml(yellow, ebay, wi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(60 Commentaires)
6.0
schoudhary1553

Hello, I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projects like this. For evidence you can see my profile. Pl Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 4 jours
(14 Commentaires)
5.8
brianconey

Hello How are you I have read your job description in detail . I am Xu It is a piece of cake i have done it before Please contact me and Let's discuss more thanks for your posting . hope to hear from you .

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(17 Commentaires)
5.5
mtanveer2012

Hi, 1. Project Price in Full to do phase 1 Ans: 150 USD. 2. Time to complete, since I am really needing this soon Ans: 1-2 Days maximum. 3. What language you will be using Ans: Python. 4. Any relevant experience you ha Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(38 Commentaires)
5.3
smsaurabhv

‌Hi, I have gone through your requirement to scrape lots of websites. I am EXPERT in building scraping tools /scripts. Hence, I can SURELY work on your project. I am having 4 YEARS of EXPERIENCE in developing PHP-PYTHO Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(51 Commentaires)
5.0
alexwmsoft

Dear Client Thank you for taking for your precious time to looking for my bid. I have fully read the job description and it is very clear to me. I am a senior web scraper who is good at python. I am using pytho Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(4 Commentaires)
4.5
sky930320

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(8 Commentaires)
4.1
shiyusun180409

I checked your description. As I am a software and scrap expert , I have a wealth of experience and skills in these projects. I prefer python language and i will complete it within 1~2 days. You will get good results. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(13 Commentaires)
4.2
shaktistanwar

Hi I am a technical evangelist, developer and can do this task for you. I have more than 15 years of experience in software development and am a microsoft certified professional.I have strong work experience in Nopcom Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(1 Évaluation)
3.4
PiushGoutam2018

Hi, I am interested in taking up your project. I have done 10+ projects in python software development,website development,desktop automation,website automation and web scraping,.I deliver fast,clean and quality work Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(17 Commentaires)
3.4
NonikNone

Hello how are you? My name is Ugur Catak. I am a software developer. About your project: Since we don't need to download entire page, we can request for the status code only to speed the bulk query cycle. 1. Project Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(2 Commentaires)
2.4
Xiong402Jin

1. Project Price in Full to do phase 1 $200 2. Time to complete, since I am really needing this soon two days 3. What language you will be using python 4. Any relevant experience you have with crawl Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(3 Commentaires)
2.3
drohanyadav

Hi, I will be using python language to do this project. I work fast, and I will probably finish it in few hours, but there are always some small complications, so it will be take me maximum of 2 days. I have been web s Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(3 Commentaires)
2.0
dhirajbaheti

Hi, My name is Dhiraj and I am a professional web scrapper and data mining expert in python with more than 6 years of experience. I have expertise in scraping complex/dynamic websites using requests, BeautifulSou Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(3 Commentaires)
2.0
frelanceralam201

Hi 1. 150$ 2. one day 3. NodeJs 4. I'm a senior developer and have good experiences on web crawling and scraping development. Although I've just started using this website, I have a lot of experience with Selenium Webd Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(1 Évaluation)
1.5
BestService222

Hi I've carefully read the description of your project. I am very interested in your project As I am a professional Crawler Developer, I am sure I can help you I'm ready to start to work on your project right now. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(1 Évaluation)
0.8