Fermé

I need a web crawler

I need a webcrawler that can find websites that are using or have Flash elements on their website.

Furthermore it needs to be able to differentiate the language of the website. I am ONLY interested in finding German and English language websites!

The project literally requires that I find ONLY websites in German and English language. That could be a whole number of different domain endings such as .de .at .ch for German language and .com .[login to view URL] .[login to view URL] [login to view URL] and so forth for English. But I definitely don't want websites that are in any other language (french, spanish, italian, polish, russian, japanese, chinese etc).

The crawler needs to generate a list of domains and their e-mail addresses for THAT language.

The crawler needs to have a GUI where the above parameters most be adjustable. It also must have the possibility to check against a blacklist, so if he hits a known domain or email, it gets skipped from the beginning.

There is a second stage where the crawler is supposed to find all websites that do not have an SSL certificate.

And a third stage to find those websites that use cookies, but do not have a cookie banner.

If the crawler can do all 3 of the above things, that much the better. But the first stage is to find websites with Flash elements.

I need a minimum of 2 million addresses per language

I am looking forward to your offers!

Compétences : Web Scraping, PHP, Python

Concernant l'employeur :
( 1 commentaire ) Kensington, South Africa

Nº du projet : #20352419

53 freelance font une offre moyenne de $1234 pour ce travail

arhamsoftltd

Hi, I can design and develop a web crawler for you. I am familiar with related libraries and frameworks such as Tensorflow, Keras, Theano, Torch Scipy, Scikit-learn. and I am skilled in Python, R, Lua, Go etc. So thi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 35 jours
(166 Commentaires)
9.0
aqualeafitsolpl

I have checked your requirements where you mentioned that you need a webcrawler that can find websites that are using or have Flash elements on their website. Please invite me in chat so we can discuss in detail.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 20 jours
(143 Commentaires)
8.3
wpoppo

Hello, As per need, you need a web crawler, we have shared our company portfolio please go through it: We have a team of professional Full Stack developers with an experience of more than 8+ years in creating and cus Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 20 jours
(186 Commentaires)
8.3
mingxiao2008

Hello, Dear How are you? I have checked your project description and am ready for discussing with you about project as Web scraping developer. I have experienced in PHP , Python, Web Scraping. I will work very hard and Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 20 jours
(104 Commentaires)
8.1
seekdeveloper

Hi, I have read your post and understood your requirement. Looking for the freelancer to work on your next project? Or just need some issues/bugs/fixes ASAP? I have 11+ years of experience and I'm here for you! My sk Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(104 Commentaires)
8.7
mingah429

Hello, I read your description and got what to do. I am looking forward to discussing more for your project. I am ready now, please message me to start your project. I have 7+ years of experience in this field I am Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(20 Commentaires)
7.9
OutsourceMan

Greetings! My name is Neha. We would love to have the opportunity to discuss your project with you. We can read through the job details extremely carefully and absolutely sure that we can do the project very well. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 12 jours
(335 Commentaires)
8.1
infowider11

Hello, How are you? Yes i am expert in web scraping . i have more than 6 year experienced in php. i can do this scraping in php, You can check my work:- [login to view URL] Thanks Shweta

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(114 Commentaires)
7.6
latatestTech

With Freelancer Preferred badge bound to give 100% Quality (Let's Chat) Hello, A great team is here for your service!! I read the project description properly, and I agreed to fulfill 100% of all your requirements. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(151 Commentaires)
7.8
ZhenExpert

Hi, I'm experienced web scraping developer with 5 years of hands-on experience. I have created several scraping projects and even developed automatic scraping bots. I'm good at python programming language and used scra Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 20 jours
(39 Commentaires)
7.0
Hariomtechnology

Hello BrandBuilder63, I have checked your project(I need a web crawler) requirements. We are one of the best programming team on freelancer. Awarded PREFERRED FREELANCER BADGE by Freelancer.com We have more than 127 Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(83 Commentaires)
7.1
darshring

Greetings! Seeing is believing - please check my recent reviews and portfolios first. I am proficient in Web scrapping and Data mining, and I have already finished several smiliar projects before. I am confident in m Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(37 Commentaires)
7.2
bestit4u

Hi. I am very interested in your project, because I have much experience in such projects. I have good skills with the program language including C/C++, C#, java, php, asp.net, python, VB.NET. So I have expert and smar Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(112 Commentaires)
7.1
C3guru

Hello I am a talented Web developer. To detect Flash elements, I can scrape Web pages from each website using selenium. In this way, we can also detect languages. The second and third stage is a bit difficult but I ca Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(28 Commentaires)
6.7
WebnAppExperts

Hello there, As you have posted jobs "I need a web crawler". We are a team of 60+ members in PHP, CodeIgniter, WordPress, Magento, Big commerce, Interspire, , Node Js, AngularJs, and Mobile Apps having experience ove Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 70 jours
(22 Commentaires)
6.3
pvsysgroupinc

Dear Sir, Requirements : - CMS AS per your need or we can make custom CMS too - Design : HTML5, CSS 3, Bootstrap - Jquery - CMS will Custom CMS - Validation Java Script - Google Map API - API ( FACEBOOK ) https:/ Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 40 jours
(24 Commentaires)
6.3
sayno2bugs

Highly experienced web scraper developer. My web scrapers scrape more than 10 Million pages each day without getting blocked. Please check my reviews here to know more about the quality you can expect : [login to view URL] Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(30 Commentaires)
6.4
Max01

Hello, I love to create Web crawler that can find out German and English language websites! using flash on their websites. I see web crawler should generate a list of domains and their e-mail addresses for THAT langua Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(251 Commentaires)
6.6
Bluesky122

Hi. I am Guicai Y. I have read your post content and I am able to help you. I am web expert using python. I have made more project using it for client. And otherwise I am web scraping expert too. I have more knowledge Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(45 Commentaires)
6.3
tekzee

Hello, Greeting for the day! I recently came across your job post and feel that I can assist you perfectly. I am a senior web developer with an experience of 8+ years, working proficiently on various domains which in Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(37 Commentaires)
6.3