I developed a Java program to scrap information from a website. The architecture of the solution involves: 1) using Java Selenium to send requests to the webpage via Chrome Webdriver to trigger authentication and authenticated requests; 2) routing the requests from Chrome (headless) to Java BrowserMobProxy to capture three HTTP headers (Authorization, X-CSRF-TOKEN, and Cookie) and one query string (without these, the server after some requests starts responding 512); and 3) use these 4 elements in HTTPs requests from Java directly to the webpage (i.e. without Selenium, Chrome, and BrowserMobProxy involved) to retrieve the desired information.
This program does the basic functionality of extracting the information but has a few problems:
It depends on an external non-Java component: Chrome WebDriver
It depends on Java Selenium and Java BrowserMobProxy, two dependencies that I would like to remove
It is not optimized (too much refresh and too long sleep periods) relatively to the limit upon which the Webpage (Cloudfare) starts responding 429 errors. Thus, the retrieval of the information is taking much more time than needed.
You will get the current program Java code and you will need to solve the problems above. To do so, you will need to:
B. You will need to identify the limit upon which the Webpage (behind Cloudfare) starts responding 429 errors. You will need to tune the refresh frequency of the headers and sleep periods to the limit identified. You will need to demonstrate the benefits of your changes by extracting the information currently extracted by the program and measuring how long it takes.
Note: you will need to create your own login/password in the webpage. No additional requirements exist to register.
8 freelance font une offre moyenne de $513 pour ce travail
Hello, I can help you to Improve webpage scrapping solution I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several projec Plus
Hello. How are you? I have already read your description and i think i am qualified for this subject. I'm full-stack web developer. "He that has most time has none to lose" it's my creed. I'll solve your anyproblems in Plus
Hi, sir. I have carefully checked your requirements and I was glad that I've already done this kind of projects before. I'd love to share more detail with you over chat and I'm sure that you'll be interested in them. I Plus
Thanks for project posting and I respect it I recently worked on the project like yours and can provide you demo work as well Do you want free demo ? ping me in freelancer message board Thanks and Regards, Plus
I live in wordpress, I can easily help you in woocommerce on anything you are trying to do, and I can even do it on screenshare if you like! Please check out my work here: [login to view URL] Rate $40hr an Plus
⭐No Problem. I can start immediatly⭐ ✔✔✔Fastest, I am sure I won't let you down.✔✔✔ ✔Before starting, I wish that you and your family are doing well in these unprecedented times. ✔I have read through your entire descri Plus
I have experience in scrapping various website pages including ajax pages without using selenium. Fine tune the frequency of refresh can minimize the error, but no guarantee of free from robot detection of your target Plus