Architecture des données Jobs

I’m ready to move my real-time data processing into Databricks and need a specialist who can build a robust streaming pipeline end-to-end. The source will be Apache Kafka; events arrive continuously and must be processed, transformed, and stored inside the Databricks environment with low latency and solid fault tolerance. You’ll design the job structure, configure Structured Streaming, handle schema evolution, and implement checkpoints so that the pipeline restarts cleanly after failures. I also want concise notebook-based documentation that shows how the pipeline is triggered, how it scales, and where metrics can be monitored in the workspace. Deliverables • A working Databricks notebook (or repo) that ingests from our existing Kafka topics, performs necessary tra...

I run several AI-driven services that rely on structured data stored in relational databases, and I want to keep them healthy, fast, and dependable. The most immediate need is continuous system monitoring and preventive maintenance: watching model performance, data pipelines, resource usage, and alerting thresholds so issues are caught long before they affect users. Alongside that oversight, I’m keen to squeeze more efficiency out of the algorithms already in production. Query plans, feature-engineering code, batch jobs, and model inference routines all hold room for improvement, and every millisecond we save lets us serve more users on the same hardware. Here’s what I would like delivered: • A baseline health audit of the current AI stack, highlighting failure points a...

I need a production-ready infrastructure that can capture and persist high-volume, real-time streaming data in a NoSQL environment, with MongoDB as the core database engine. The cluster must be architected for horizontal scalability, high availability, and low-latency writes so that, down the line, I can mine the stored data for machine-learning model training without worrying about bottlenecks or data loss. Here’s what I’m expecting: • A detailed architecture plan that explains shard keys, replica-set topology, and how the cluster will expand as data grows. • Deployment of the MongoDB cluster (on-prem, cloud, or hybrid—whatever we decide is best) scripted through infrastructure-as-code so the environment can be recreated in a single command. • A stre...

Microsoft Fabric Pipeline & Notebook Ownership / Service Principal Migration We are looking for expertise to help us migrate the authentication and execution model of our existing Microsoft Fabric Data Pipelines, Notebooks, Connections, and Scheduled Triggers from individual user accounts to a Service Principal. Currently, several Fabric pipelines and notebooks are created, owned, and executed using individual user identities. Scheduled triggers and pipeline executions are also dependent on these individual accounts. The key challenge is user dependency and ownership continuity. When an employee who created or owns a pipeline/notebook leaves the organization or moves to another role, the associated Fabric artifacts and scheduled processes may continue to be tied to that individual&#...