Data Engineering Consultant

@FreeLancers360

Hi there, Your project "Data Engineering Consultant " instantly caught my attention — it's exactly the kind of work we love to take on. I love turning a solid brief like yours into a polished result you’ll be proud to put your name on. From your brief I can see this involves node, frontend — all areas we handle in-house. We specialise in SQL, Data Warehousing, Docker, Data Architecture, which lines up directly with what you need. How we'd approach it: - Map out the scope, pages and data models up front - Build the frontend and backend with clean, scalable code - Test across devices and browsers, then deploy - Hand over with documentation and post-launch support Happy to work hourly with transparent time tracking and regular check-ins. Expect smooth progress updates throughout, full respect for your specifications, and zero surprises along the way. Send over any references you have and I’ll come back with a clear plan and milestones. Best regards, FreeLancers360 Let’s connect in chat and get started — message me anytime and I’ll reply right away!

@vishal1145

Your lakehouse architecture will fail under concurrent writes if Iceberg table maintenance isn't automated with proper compaction schedules. Without this, query performance degrades exponentially as small files accumulate. Quick questions - what's your current data volume and ingestion frequency? And are you running Spark on Kubernetes or EMR? Here is the architectural approach: - APACHE SPARK: Design PySpark jobs with dynamic partition pruning and adaptive query execution to handle your event processing at scale without memory spills. - DATA WAREHOUSING: Implement medallion architecture using Iceberg's time travel and schema evolution features so your bronze/silver/gold layers stay consistent during backfills. - DOCKER: Build multi-stage Dockerfiles with health checks for Airflow workers and Spark drivers to prevent silent container failures in production. I've architected similar lakehouse systems for 2 fintech clients processing 50M+ transactions daily. Let's schedule a 15-minute call to review your data flow before you commit to the build.

@ParasTech7

Dear Hiring Team, Hope you are doing great! I am a BTech from IIT and have been doing all these things for many companies in my career that spans 20+ years. I can do this for you as I am very good at Distributed Systems and related technologies :) Choose me if you like! More once you revert. Regards, Anurag.

@indreshpnk

Hello, I understand you need a data architecture solution covering distributed event processing, Spark/PySpark, Iceberg lakehouse, Airflow orchestration, analytical SQL, and Docker. The goal is to build reliable, scalable pipelines that are maintainable and production-ready. Here’s what I can provide: Design Bronze/Silver/Gold lakehouse pipelines using Apache Spark, PySpark, and Iceberg Build batch/streaming workflows with Airflow, SparkSubmit, drivers, and robust monitoring Develop analytical SQL, Dockerized services, APIs, testing, and scalable data-processing components I bring over 4+ years of experience in Python, SQL, Docker, REST APIs, databases, and full-stack development, with a strong focus on scalable architectures and reliable data-driven applications. Just to clarify a few things: Is the existing data source primarily transaction/GST/banking data, or multiple sources? Do you already have an Iceberg/lakehouse environment and Airflow setup, or should I design it from scratch? Please come to the chat box to discuss more about your project. Best regards Indresh Kushwaha

@Bhavyan761

I’m a Data Science/ML engineer with strong hands-on experience in Python, SQL, data preprocessing, feature engineering, machine learning pipelines, and backend/API development, and I’m comfortable working with data-heavy systems. Your project particularly interests me because it combines distributed data processing, analytical SQL, data architecture, and entity matching. I can contribute across: - Apache Spark / PySpark for distributed processing and large-scale transformations - Analytical SQL, including CTEs, joins, aggregations, and window functions - Data warehousing / Lakehouse concepts, including Bronze/Silver/Gold pipeline design - Python for APIs, automation, data processing, and testing - Entity matching / record resolution, including similarity-based matching and precision/recall evaluation - Docker for reproducible development environments and services - Data pipelines involving ingestion, transformation, validation, and monitoring - Graph/tree concepts such as nodes, edges, hierarchies, DAGs, and adjacency structures I focus on building solutions that are maintainable and production-oriented rather than just getting a pipeline to work once. I’m also comfortable understanding an existing architecture, identifying bottlenecks, and improving individual components without unnecessarily changing the whole system. I’d be happy to discuss the current architecture, expected data volume, Spark workload, and the specific entity-matching requirements before starting.

@prachia77

Hi, This project aligns well with my backend/data engineering stack. I can work with Python, SQL, Docker, APIs and data pipelines, with a focus on building maintainable ingestion and processing workflows. I understand the importance of separating bronze/silver/gold layers, reliable orchestration, analytical SQL and production monitoring. For the Spark/Airflow/Iceberg-specific components, I’d first map the existing architecture and then implement the required pipelines, containerized services and data-quality checks in a modular way. I can also integrate supporting APIs or search components where required. Best Regards, Webix Infotech India Pvt. Ltd.

@alfatrade2

Your list mixes Spark internals with Iceberg medallion layers and Airflow, and that usually means the real question is where the current pipeline breaks: skew, small files or orchestration. I haven't run Iceberg in production, so I'd say that upfront. What I bring is the pipeline side: I built an AI classification pipeline for tender monitoring with isolated per-source adapters. What is the pain point today, and is it batch or streaming?

@LeonBoshoff

I specialize in data engineering and have recently tackled similar projects involving Apache Spark and Lakehouse architecture. I understand the intricacies of distributed event processing and the importance of efficient data workflows. Your need for expertise in Apache Airflow and SparkSubmit for streaming and ingestion pipelines aligns well with my experience. I can design robust data architectures independently and ensure they perform optimally in production. I emphasize clear communication and a practical approach, ensuring that the solutions align with your business goals. My background in financial services helps me understand the nuances of transaction data, which is crucial for your project. What specific challenges are you facing with your current data architecture? If you choose not to work with me, you still get a free consultation from an expert in the field. P.S I look forward to your message Leon B.

@dineshkumar1311

Hi, Your post reads like a checklist of my daily stack. Here's exactly how I'd deliver it: Spark/PySpark — Streaming + batch pipelines, tuned at the execution-internals level for heavy financial loads. Lakehouse — Medallion Architecture (Bronze/Silver/Gold) on Apache Iceberg with ACID + schema evolution for transaction data. Orchestration — Airflow driving SparkSubmit and custom drivers across ingestion and streaming. SQL — Window-function-heavy analytics for fraud and reporting. Docker Compose — Services with health checks, production-ready. Ownership — I design the architecture independently and monitor it in production with freshness/health observability. Good-to-have, covered: Zingg-style entity matching + golden records with precision/recall, Python APIs/testing, Elasticsearch indexing, graph/tree structures with D3.js/Cytoscape.js, Angular + NgRx/Akita, CI/CD. Domain: fintech, transactions, GST, banking, fraud. My first 3 steps: Audit your ingestion pipelines and Medallion/Iceberg setup. Build idempotent, replayable Spark pipelines via Airflow, containerized. Own production with clear observability metrics. One question so I can scope accurately: is your primary pipeline streaming or batch, and roughly what daily data volume are we talking? Happy to walk you through relevant work on a quick call. Best, Dineshkumar

@CharlSche

Just completed a similar project focused on distributed event processing using Apache Spark and PySpark, ensuring a seamless data flow. The architecture we implemented was robust, leveraging lakehouse principles, which aligns perfectly with your needs. My expertise includes designing and overseeing data architectures, particularly in data warehousing and workflow orchestration with tools like Apache Airflow. I have hands-on experience with analytical SQL and Docker Compose for container management, ensuring high performance and reliability in production environments. I would love the opportunity to discuss your project in further detail. Best Regards, Charl

@kemalakra

Hi, I’m Muhammad Akmal, a Senior Software Engineer with 13+ years of experience in SQL, relational databases, backend APIs, enterprise architecture, performance troubleshooting, CI/CD, and production support. My strongest experience is in .NET/SQL-based enterprise systems rather than Spark/PySpark lakehouse engineering, so I would be best suited to database, SQL, API, integration, architecture review, and production-support aspects of the engagement. I’m comfortable analysing existing systems, improving data access and query performance, designing maintainable backend components, and documenting deployment and operational workflows. If the role requires deep hands-on ownership of Spark internals, Iceberg, Airflow and large-scale lakehouse architecture from day one, I would prefer to clarify that before proceeding.

@JoseRCarvas

Hi — Data engineering: I work with Spark/PySpark distributed processing and can take on the runtime/execution issues and pipeline work in your spec — clean, tested and documented. I hold a 5.0-star rating here and deliver clean, tested, working code. We can start with a small funded milestone so you see real progress before committing further. Ready to start today — happy to talk through the details. - Ricardo

@sachink973

Hi, I can help with this project. I have 3+ years of data engineering experience with **PySpark, Apache Spark, Airflow, Databricks, SQL, AWS/GCP, and Docker**. I’ve worked on production-style ETL pipelines, Lakehouse/Medallion architecture, Spark transformations, and Airflow orchestration. I can review your current architecture/code, identify the issue or bottleneck, and implement the solution hands-on. I’m available to start immediately. Could you share the current Spark/Airflow architecture or the first issue you need solved?