We want to highlight buildings that have apartments through Mapbox. Each of these highlighted buildings should have information about the apartments in it (the one we get from our working scraper which we should remake by adding some new fields). Mapbox will highlight apartments via a certain parameter, in our case this parameter is av = 0 and av = 1 (av - availability), where av = 0 - there are no apartments in the house, it is not highlighted it and av = 1 where there are apartments should be highlighted.
What we need:
A database (we think of SQL, but there may be better options), which will store the following data: coordinates of the house (in which the apartment is), address, links to the apartment on [login to view URL], description of the apartment (need to be added to the scraper), price, area, number of bathrooms and bedrooms, links to images.
We have one big .geojson file with all the buildings in New York City. It serves as the system layer in Mapbox. This file contains parameters such as PID (polygon id), height (for 3d buildings), coordinates of the polygon and the parameter av = 0 (because buildings are not highlighted).
We have a working scraper. It should scrape the necessary information and output it in the format .geojson, with the addition of the parameter av = 1 (because such buildings should be highlighted). The keys in the scraper should be instead of Number of bedrooms key changed to Number_of_bedrooms. When this file is ready, it should be uploaded to the database. In the database it is necessary to synchronize two .geojson (system and scraped one), that is, fill apartments info in the empty rows of the corresponding coordinates, and therefore it is necessary to check whether a certain apartment is not already in this base (the base should be updated every 4 hours, it is necessary to check coordinates and other info whether this apartment is in the DB. As the DB already has a system layer, which contains information about the polygons (ie the buildings themselves) you should not forget that the scraped info doesn't contain polygons info, it carries only the coordinate of the building, so you need to check which of the polygons is the coordinate of the scraped apartment and add the information we need to the database. There are two options: inside() function or others which allow us to check if a certain coordinate is in a certain polygon (in this case it is necessary to check in our system .geojson, there are all polygons) and the second option: our own table "polygon-coordinates": each polygon corresponds to certain [login to view URL] on this table should be faster than programmatically comparing coordinates with all polygons each time (and the .GeoJSON system contains more than 1 million buildings and weighs 400+ mb). We have a csv file that has list of addresses and 700k+ coordinates. So you should make a small scraper with the function inside () or analogue and output it all in one base, in which each coordinate corresponds to a specific polygon. There may be cases when not all the coordinates of all the NYC buildings are in the csv file, which means that when the such data arrives, it must be checked first in the "polygon-coordinates" table and then (if there are no matches), check them separately on the function inside () or analogues. When it's over, the database should be like this: there are many polygons (system layer + those highlighted). The system layer polygons have av=0 parameter, the highlighted ones have av = 1 and information. The database then needs to export it all to a new .geojson file. This file will be updated every 4 hours as scraping will take place every 4 hours. Therefore, it is important to keep in mind that you should constantly check whether a particular apartment is already in the database (starting with the coordinates, description and ending with the price, as there is often a price update or 5 apartments in the same house).
Each polygon corresponds to only one building.
20 freelance font une offre moyenne de $190 pour ce travail
Hi there,I am Web Scraping expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this project ! I can start immediately and finish it within the Plus
Hi.Dear... I can scrape anything.I know all the angles of website scraping. I have much experience for 10+ years such as avoiding recaptcha, using multi threading, avoiding site security.... I can do your job with craw Plus
I have lots of experience writing web automation scripts. Available to start immediately and finish as soon as possible. Please contact to discuss details if you are interested. Looking forward to work on this project. Plus
Dear Sir, I am interested in your project. I have gone through your requirement. I'm expert in web scrapping and web automation using selenium and jsoup, data management, data mining. I assure, I will do my best to w Plus
Hello, After reading your project details I believe I'm suitable for this project. As I'm expert on it with more than 7 years experience. Please feel free to contact me. I am looking forward to hear from you. Plus
Hi Dear! Today I have read your job, I am very interested it and think this project is for me. Web scrapping is my base skill. I am a senior development and have rich data scrapping experience in various Websites like Plus
Hello\nI am a python developer.\nI have great experience in web scraping and I am an expert in it.\nI have all necessary skills by which I can scrape any website.\nPlease message me to discuss in detail.
Greetings I have 3 years of experience in python, web scraping. I have scraped more than 100 websites. I can help you scrape this website . I have looked at this website. Ping me for further discussion
hello sir, i worked on several scrapping projects, i read all the description you write, and i understand what you want, i just need to discuss with you some details. i am available, ready and highly motivated to work Plus
Hi. Please survey my reviews. I saw your description carefully and I am confident I can full fill your requirement. So I provide my skills in hopes of working with you. I am full-stack developer and I have 2+ years Plus
Hello, I have Experience with scraping Websites like FB ,IG and Telegram and Experience in Google APIs. I have 5 years of experience with Python. I have worked with many libraries in python for tasks such as Data Analy Plus
Hello, I am an experienced developer and coder with very good experience in data scraping. I can make the scraper exactly as per your requirements. As the data set contain a large amount of data using SQL type databa Plus
Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages, data entry. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, Plus
I have a good scraper already produced for other projects that can easily be customized for this specific use. I also have a business intelligence database system that I developed as an open source project that can be Plus
Hi, I already created the scraper which will scrap all these details from trulia site. Kindly message me if you need this. Thanks
Good day! This is my woocommerce shoping site. [login to view URL] I have developed it from scratch to the end. and I have many experiences in laravel, wordpress, CI, python as backend. And I had several experiences in angu Plus
Greetings! My name is Daniel and I am a software developer from Germany, specialized in python applications. I have finished many web scraping projects and after looking at your attached flowchart, I am quite confident Plus
Dear sir. I think you need web scrapping for your data. It's easy. I will suport you by using IronWebScraper-C# framework or NodeJS-Nightmare Library , Dataminer Scraper, and Python in your favourite ways. I have full Plus
Dear Sir or Madam: I am doing crawling, scrapying and ETL for a long time, many proyects.I have also worked with geolocation API of Google. According to your description a succesful system could be achieved within a m Plus