Fermé

Improve webpage scrapping solution -- 2

I have a Java program to scrap information from a website. The architecture of the solution involves: 1) using Java Selenium to send requests to the webpage via Chrome Webdriver to trigger authentication and authenticated requests; 2) routing the requests from Chrome (headless) to Java BrowserMobProxy to capture three HTTP headers (Authorization, X-CSRF-TOKEN, and Cookie) and one query string; and 3) use these 4 elements in HTTPs requests from Java directly to the webpage (i.e. without Selenium, Chrome, and BrowserMobProxy involved) to retrieve the desired information.

This program does the basic functionality of extracting the information but has a few problems:

It depends on an external non-Java component: Chrome WebDriver

It depends on Java Selenium and Java BrowserMobProxy, two dependencies that I would like to remove

It is not optimized (too much refresh and too long sleep periods) relatively to the limit upon which the Webpage (Cloudfare) starts responding 429 errors. Thus, the retrieval of the information is taking much more time than needed.

Deliverables

You will get the current program Java code and you will need to solve the problems above. To do so, you will need to:

A. Find out how to authenticate and refresh the 3 headers and the query string without depending on Selenium, Chrome Webdriver, and BrowserMobProxy. As most of this data is likely generated in JavaScript, you will need knowledge about JavaScript and how to execute JavaScript from within Java or convert the JavaScript code to Java (preferable solution).

B. You will need to identify the limit upon which the Webpage (behind Cloudfare) starts responding 429 errors. You will need to tune the refresh frequency of the headers and sleep periods to the limit identified. You will need to demonstrate the benefits of your changes by extracting the information currently extracted by the program and measuring how long it takes.

Note: you will need to create your own login/password in the webpage. No additional requirements exist to register.

Compétences : Java, Web Scraping, JavaScript, Python, Architecture Logicielle

en voir plus : scrapy, web scraping, web scraping python, web scraping techniques, web scraping python beautifulsoup, web scraping tutorial, web scraping algorithm, web scraping tools, webpage pdf solution, solution counting words text file finding frequency write test mips assembly language program, how much does it cost to hire a scientist for 2 hours, how long does it take to edit a 2 minute video, how long does it take to migrate from magento 1 to magento 2, how long does it take to improve credit score, how long does it take to register a trademark in south africa, how much does it cost to register a company in usa, csr racing 2 jaguar f type tune, how long does it take to write a 2 page paper, how much does it cost to go to australia for 2 weeks, yii 2 register script

Concernant l'employeur :
( 1 commentaire ) Băilești, Romania

Nº du projet : #26802689

5 freelance font une offre moyenne de $177 pour ce travail

webprof2

Hi, Please see my previous project done in Java. I'm expert in Java and have good knowledge in scrapping methods using selenium. I', also fully familiar with JavaScript (client and nodeJS server side) and so may use t Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 10 jours
(84 Commentaires)
5.5
AugmeticTech

Hi there, I am a scraping expert and I can start this task right now. Let's connect for intro. Regards, Sandip, Augmetic Technology [login to view URL]

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(1 Évaluation)
2.5
%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(0 Commentaires)
0.0
kminors1

I Specialize in API scripts. Have worked on financial data, eCommerce sites, B2B sites, real estate data. Have 150+ 5/5 star reviews on another platform for API related work and very good with captcha work..

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 6 jours
(0 Commentaires)
0.0
NikitaYas

Good day. I'm interested in your project. I have about 3 years of scraping and 6 years of python programming experience. A big plus of using python is that everything will be automated and that I can write a program qu Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(0 Commentaires)
0.0