Find Jobs
Hire Freelancers

Kafka & Elastic Real time data pipeline & API

$250-750 AUD

Fermé
Publié il y a presque 4 ans

$250-750 AUD

Payé lors de la livraison
I need a data pipeline designed using S3, Kafka, Spark, and Elasticsearch / Snowflake. Something a lot like what is described in this Esri presentation [login to view URL] - however a lot simpler as we will do a basic spark transformation (not build a complex library of batch and real-time transformations). The process is that: JSON files will be dropped into S3, these will be instantly picked up by Kafka and Processed by Spark, then this processed data will be pushed into elastic ready for access via the HTTP api and also into snowflake for querying later. Don’t mind if there’re are any suggestions on how to do this better. Prefer using serverless products AWS or GCP. Fine with either Python or Java. The final deliverable is the data pipeline with documentation, and a 2hr video conference training session with me (experienced programmer) who you will run through the solution with and explain all questions. Please only respond if you have experience building data pipelines with the mentioned technologies.
N° de projet : 25775298

Concernant le projet

14 propositions
Projet à distance
Actif à il y a 4 ans

Cherchez-vous à gagner de l'argent ?

Avantages de faire une offre sur Freelancer

Fixez votre budget et vos délais
Soyez payé pour votre travail
Surlignez votre proposition
Il est gratuit de s'inscrire et de faire des offres sur des travaux
14 freelances proposent en moyenne $581 AUD pour ce travail
Avatar de l'utilisateur
Hi mate, I am AWS Certified developer with programming background and currently working full time as a Cloud DevOps engineer. Please see my ratings for my past customer focus and project delivery performance. I will make sure the solution is serverless, scaleable, highly available, and secure as per Well Architected Framework. For language i'll go with python and happy to do a KT session. Let me know if I can be of any assistance. Thanks Regards, Salman
$500 AUD en 15 jours
5,0 (178 commentaires)
5,9
5,9
Avatar de l'utilisateur
Hey there, I have 4 years of working experience in web technologies and I have a strong working knowledge of every possible framework which is in the market. I know: Amazon Web Services, Python, Software Architecture, Java and Google Cloud Platform Also I have done tons of work on freelancing career. I am waiting for a positive response from your side. And I have read your requirements as well. Thanks and regards, Ayesha
$750 AUD en 24 jours
5,0 (6 commentaires)
5,2
5,2
Avatar de l'utilisateur
Hello! I'd like to deliver analytics solution. I'm familiar with amazon web services, pyspark. I'll do the job fairly good. Please, give me a try!
$250 AUD en 7 jours
5,0 (10 commentaires)
5,2
5,2
Avatar de l'utilisateur
Hi, I am a java developer having 9+ years of experience, I understand your requirements very clearly, I have good real time experience in elastic search, kafka , spark. I am a permanent employee in Oracle and I have involved in a project having all these technologies. I am confident to complete it withing 2 weeks. Happy to develop with local server or hosting server also instead of aws. My skills: I am well experienced in Java and J2EE. Good UI experience in PHP, JSP, HTML and AJAX Good experience in MySQL database Good experience in kafka, elastic search , kibana, spark Great knowledge about software architecture and design pattern. Very much excited about your project, I am happy to discuss about it. I have some good suggestion that I have seen in my professional career, let's talk in details. Regards, Prakash
$500 AUD en 14 jours
4,9 (35 commentaires)
5,0
5,0
Avatar de l'utilisateur
Hi, I am a certified developer in this array and I can deliver you the best quality work. Please let’s connect and discuss more on your requirements. Thanks, Naresh.
$556 AUD en 30 jours
5,0 (7 commentaires)
4,4
4,4
Avatar de l'utilisateur
Hi Dear. Very pleased to meet you, who are interested in Big Data, Elastic Search. I have a special way with Big Data, especially Data Analytics & Data Indexing & Search using Apache Solr. Of course, You know Apache Solr is one of the most popular Bid Data Techniques. Lately, I've worked on the medicine project about Elastic Search & Semantic Search using Apache Solr. Let me send demo site url. Here, I implemented Full text indexing & Elastic Search & Semantic Search & Document Ranking & Faceting & Document Recommendation. I hope long-term. Hope to see you soon
$500 AUD en 7 jours
5,0 (2 commentaires)
2,1
2,1
Avatar de l'utilisateur
Hello, I understand your project completely. I watched that youtube video. It is amazing to see their stack. I can set up this stack totally on Cloud. Experienced in elastic search, logging and monitoring, kafka, AWS cloud and little hands-on spark. But yes I have many professional with me who have experienced on spark so I will handle that part also. Please connect with me and we can discuss it more. I am a DevOps Engineer having years of experience in DevOps field. My proficiency is in managing whole DevOps Lifecycle for any project. My expertise fields are :- AWS Kubernetes Docker ECS CI/CD(Jenkins) AutoScaling and Load balancing Setup Configuration of H.A. Infrastructure ECS Setup logging and monitoring tool Terraform Cloud formation GitHub Ansible Please Direct message me for further conversation and we can discuss your project in brief.
$300 AUD en 7 jours
5,0 (2 commentaires)
2,1
2,1
Avatar de l'utilisateur
Hi- I am senior developer and architect with over 15 years of experience in java/j2ee, scala/akka, big data and Machine learning, AWS, Google Cloud, Azure cloud technologies. For building this pipeline, we can use S3, Kinesis (Equivalent to kafka) provided by AWS, then by lamda or EMR based spark job then to Elastic Search or Snow flake. Please, let me know if this suits your requirement and let me know if you need any further details. Regards Ramakrishna SCJP5/SCBCD5/SCWCD/SCMAD/SCDJWS/GSSP
$500 AUD en 15 jours
0,0 (0 commentaires)
0,0
0,0
Avatar de l'utilisateur
GHAARVIT SOLUTIONS team have experts in Hadoop Bigdata technologies mainly in Spark(using Scala),Kafka (Using Java), KSQL and Elastic Search collaborating with AWS. Please consider this as a proposal for 750 AUD. We can negotiate the final figures after reviewing the full requirement. Note : We provide 1 month FREE warranty support for all our projects.
$750 AUD en 20 jours
0,0 (0 commentaires)
0,0
0,0
Avatar de l'utilisateur
With over 7.5 years of experience in Big Data Development, Data Engineering and Data Ingestion, I am highly capable in creating real time data pipelines using Apache Kafka, Apache Spark with Python (Pyspark) along with Scala. I am very comfortable at developing data pipelines using Pyspark, Scala and Java and have also setup and managed Spark, Hadoop and Kafka Clusters at production level. When it comes to Spark PaaS, I have great command over Data bricks and its integration with both Azure and AWS. I excel in writing clean, readable and highly understandable code follow unit test first development methodology. For CI/CD, I have worked with devops using Teamcity, Artifactory and Azure Devops. I am a Microsoft Certified Azure Data Engineer Associate and have served in Technical Lead, Senior Consultant for fortune 500 companies in the North American region. Hoping to hear from you soon around your requirements. I am happy to go over your requirements and share my experience on a phone call with you. Thanks Sameet
$750 AUD en 7 jours
0,0 (0 commentaires)
0,0
0,0

À propos du client

Drapeau de AUSTRALIA
Chippendale, Australia
5,0
2
Méthode de paiement vérifiée
Membre depuis sept. 28, 2015

Vérification du client

Merci ! Nous vous avons envoyé un lien par e-mail afin de réclamer votre crédit gratuit.
Une erreur a eu lieu lors de l'envoi de votre e-mail. Veuillez réessayer.
Utilisateurs enregistrés Total des travaux publiés
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Chargement de l'aperçu
Permission donnée pour la géolocalisation.
Votre session de connexion a expiré et vous avez été déconnecté. Veuillez vous connecter à nouveau.