Scrapy Jobs

We would like a large number of pages extracted from a website. We have a detailed spec and exact deterministic plan on how you would achieve this. You would be running the initial bot to get previous pages and then running it ongoing and providing us with the code and data.

No worries, that's way more workable. Here's the full post trimmed to fit under 10,000 characters: Nationwide Property Auction Web Scraping & Intelligent Alert System (Ongoing) About Us We're a commercial real estate investment firm that acquires distressed properties nationwide. We have the capital to close on any deal in the U.S. — our bottleneck is finding opportunities before competitors. We're building an automated system that monitors every property auction source in the country, filters against our criteria, and alerts us only on qualified deals. This is not a data dump project. We don't want spreadsheets with thousands of rows. We want a smart radar system that scans everything, filters ruthlessly, and only pings us when something matches. Long-t...

I need a Python-based solution that automatically gathers companies and shareholders data, pulls supplementary details via external APIs, and outputs a clean, unified dataset I can query at any time. Scope of the scrape • Sources: company websites, financial databases and relevant public records. • Website focus: company profiles, turnover figures and any available Demat / share-holding particulars. What the tool should do 1. Crawl or call the above sources, respecting and rate limits. 2. Parse the required fields, normalise names and IDs, then enrich each record through one or more APIs (for example OpenCorporates, Clearbit or any better suggestion you have). 3. Store results in a structured format (CSV plus an SQLite or Postgres option). 4. Offer a simple comma...

I need a reliable script or windows-application that automatically gathers text content from specified websites and online databases, then saves everything into a clean, well-structured CSV file. A Windows-software would be preferred. The crawler should be able to crawl the website and spider a list of urls for approval or automatically go through the website Or just scrape a given list of urls (from a txt-file) Key details • Sources: public-facing websites and shops (also with login using username:password) • Data type: text only—no images or binary files. • Output: one CSV per run, UTF-8 encoded, with a header row • should be able to read/exrtract data from !! various shops & websites !! -> generally i need a basic software + "plugins" fo...

We are looking for an experienced developer who can build an automated system to extract daily newly incorporated company data from the MCA (Ministry of Corporate Affairs) website – https://www.mca.gov.in. The system should automatically collect and deliver the list of companies incorporated each day in structured format (Excel / CSV / API / Database). Scope of Work: Develop a web scraping or API-based solution to extract daily incorporated company data from the MCA portal. The tool should automatically fetch newly incorporated companies every day. Data should include the following fields (minimum): CIN Company Name Date of Incorporation ROC (Registrar of Companies) State Company Type (Private Limited / LLP / OPC / Public Limited) Authorized Capital (if available) Regist...

We are looking for an experienced developer to build a robust web scraping solution capable of extracting structured data from a login-protected medical/drug repository website. The platform contains a large database of drug information (potentially hundreds of thousands to over a million pages). The scraper should be able to navigate through the website after login, systematically extract relevant drug data, and store it in a structured format. Scope of Work: Develop a scraper that can log into a protected website. Navigate through the drug repository pages. Extract structured information from each drug page. Handle pagination and large-scale crawling. Implement mechanisms to prevent crashes or interruptions during long scraping runs. Store extracted data in a structured format such as ...

I need a Python-based solution that automatically gathers companies and shareholders data, pulls supplementary details via external APIs, and outputs a clean, unified dataset I can query at any time. Scope of the scrape • Sources: company websites, financial databases and relevant public records. • Website focus: company profiles, turnover figures and any available Demat / share-holding particulars. What the tool should do 1. Crawl or call the above sources, respecting and rate limits. 2. Parse the required fields, normalise names and IDs, then enrich each record through one or more APIs (for example OpenCorporates, Clearbit or any better suggestion you have). 3. Store results in a structured format (CSV plus an SQLite or Postgres option). 4. Offer a simple comma...

I need product details captured from a set of websites and delivered in a clean, structured format I can load straight into Excel or a database. The job involves visiting the URLs I provide, pulling every product’s name, price, SKU, description, and any other specifications that appear on the page, then handing everything back to me in a .csv or similar flat file. A lightweight script—Python with BeautifulSoup, Scrapy, or a comparable tool—would be ideal so I can rerun the extraction whenever the catalogue changes, but I’m happy to discuss whether you deliver only the compiled dataset or include the code as well. Please keep the workflow ethical (no site overload, respect where applicable) and ensure the final data set is complete, deduplicated, and readable wi...

I have a curated list of specific company websites and I need an automated solution that extracts complete contact information from each one. The goal is to turn every URL into a clean, ready-to-use lead. WEBSITE : The scraper should capture: • Email addresses • Phone numbers • Mailing addresses • LinkedIn profile link • Location (city / state / country) • First and last name • Occupation / job title • Company name • Company website A well-structured CSV or Excel file is the preferred output, with each field in its own column. I am comfortable with your choice of tech—Python with BeautifulSoup, Scrapy, or Selenium are all fine—as long as the script runs reliably and respects and rate limits where required. Ac...

I need a small, always-on scraper that keeps an eye on a popular second-hand marketplace and alerts me the moment any Electronics listing matching my keywords appears. My priority is speed—ideally I hear about a new post within seconds, certainly no longer than a minute after it goes live. Here’s what the script must do: • Crawl the marketplace continuously without being blocked, parse every new listing, and filter it against a configurable set of electronics keywords. • Extract and store the Price and Condition fields so I can track changes and avoid duplicates. • Push an instant notification (email, SMS, or Slack—whichever you prefer to wire up) each time a fresh match is found. I’m comfortable with a Python 3 stack—think Requests/...

PROJECT TITLE Web Scraping Developer for Global Legal & Regulatory Data Collection PROJECT OVERVIEW We are looking for a developer who can build an automated system to collect legal and regulatory documents from multiple global sources. The goal is to create a scalable automated pipeline that can gather legal data across multiple jurisdictions and regulatory domains. DATA COLLECTION SCOPE The system will collect information related to: - Medical law and healthcare regulation - Medical advertising regulation - Corporate formation and company governance laws - Investment regulation (stocks, cryptocurrency, real estate) - Tax law and administrative tax rulings - Beauty and cosmetic regulation - Medical and cosmetic manufacturing compliance - Import and export law - Customs and tariff...

Thanks for looking. I urgently need data for a set of local businesses in and around Berkshire and London UK. We will pay per 2k list of the industries we will send upon acceptance. Email addresses must not be role based or trip any spam traps. I need a one-time extraction of verified email addresses from reputable online business directories. No other data fields are required—just the clean list of emails. Please choose whatever approach you prefer—Python with Scrapy/BeautifulSoup, browser automation with Selenium, or a similar tool chain—as long as the result is accurate and the scraping respects each site’s terms of service and rate limits. Deliverable • A CSV or XLSX file containing every unique email address you capture, de-duplicated and ready ...

I need a robust yet easy-to-maintain web scraper that pulls player statistics from four different sports sites—a blend of official league pages, sports news outlets, and a couple of well-known fan forums. All scraped data should flow into a single database and surface through a lightweight web dashboard where I can search by player, season, and team, compare numbers side by side, and export results to CSV. My ideal flow looks like this: enter or schedule the URLs, run or auto-run the scraper, watch progress logs, and then immediately view fresh stats inside the dashboard—no command-line work once everything is deployed. If any source changes its HTML, the scraper should fail gracefully and flag the issue in the UI so I can react quickly. Tech stack is flexible; Python with Be...