Fermé

Software Engineer I

Develop and validate solution architecture to support business requirements

Develop Spark applications to read transaction data and process business rules to report errors and transaction summary

Load and transform large sets of structured and semi structured.

Read data from Kafka and process data using Spark.

Adopt innovative architectural approaches to leverage in-house data integration capabilities.

Analyze existing processes and prepare functional and requirements documents.

Develop multiple spark streaming and core jobs with Kafka as a data pipe-line system.

Hive external tables on HBase using Insert Overwrite with S3 as data storage.

Troubleshoot developed Spark jobs.

Manage and review Yarn Application Logs, Spark Event Logs and Metrics sink CSV files.

Improve the performance and optimization of the existing jobs in Spark.

Develop solutions by utilizing commercial and open source software including Minifi and Nifi to interface big data and relational solutions.

Compétences : Hadoop, Java, Architecture Logicielle, Spark, Python

Concernant le client :
( 0 commentaires ) Los Angeles, United States

Nº du projet : #23958059

16 freelances font une offre moyenne de 45 $/heure pour ce travail

helmot

Hello. I have a masters degree in AI and have worked with Hadoop, Spark, YARN, ... on a lot of projects. I can share some demos if you are interested. Thanks, Helmot

$35 USD / heure
(221 Commentaires)
8.2
umg536

Hi there, Being an expert in Python and matlab programming I can do this project for you. please leave a message on my chat so we can discuss the budget and deadline of the project. Also, have a look at my profile for Plus

$50 USD / heure
(22 Commentaires)
6.3
stuartschade

Hi, I have readymade video streaming software. If you want to get the demo. I will give a demo. I have been working for the last 10+ years in development. Please have a look at my [login to view URL] Skills and Experience Plus

$43 USD / heure
(1 Évaluation)
4.9
jinc772

Hi, there. I am a senior Python & Web developer and my main skills are Django and Flask. Fully experienced in each step of web development - design, development, deployment and maintenance, and mastered management of Plus

$40 USD / heure
(5 Commentaires)
4.9
(2 Commentaires)
4.5
dineshrajputit

Hi, I have 9 years of experience and working on hadoop, spark, Kafka, nosql, java, BI tools(tableau, powerbi), cloud(Amazon, Google, Microsoft Azure)... Done end to end data warehouse management projects on aws cloud Plus

$50 USD / heure
(7 Commentaires)
4.0
fabienbenoit1984

Hello! I'd like to optimize information aggregation. I'm familiar with python and theory of probabilities. I'll do the job blazingly fast. Please, give me a try!

$56 USD / heure
(3 Commentaires)
4.0
asjkim

Highly effective Data Engineer and Data Scientist with over 25 years of experience. I am specilaizing Big Data and have extensive Python / Spark experience both in PySpark and Scala also in MLLib. I developed a number Plus

$55 USD / heure
(2 Commentaires)
4.0
dinesh4nitb

Hello, hope you are well there, I've seen your requirements and I am interested in this. I'm ready to work from now on your app. I'm a full-stack developer. I've 7+ years of working experience in Mobile Apps Developm Plus

$35 USD / heure
(12 Commentaires)
3.7
nmogilip

I am currently working on each item which you described. I delivered end to end solution including kafka, spark, hive, nifi and all other required hadoop eco system components. Please let connect or have a call to di Plus

$44 USD / heure
(2 Commentaires)
2.2
andysteiner

Hi there, I have quite some experience in most of the big data tech out there (Hadoop, Spark, Kafka, Cassandra, ...), since I have been specializing on Data Management, Data Consolidation etc. for quite a long time, no Plus

$50 USD / heure
(0 Commentaires)
0.0
Beyaz1t

I have developed country wide log parsing application using Apache Spark on Hadoop. I have experienced memory issues deeply. Sqoop Metastore Oozie workflow pipelines were used. Machine Learning is another side of thmy Plus

$43 USD / heure
(0 Commentaires)
0.0
acka3a5

I saw your posting and thought we might be a good fit. We are a full stack development firm with a wide range of experience in development. Our team is highly capable and have worked with companies including Airbnb, PN Plus

$56 USD / heure
(0 Commentaires)
0.0
LDaniel194

Hello, I read your project description and I think I'm fully qualified for this position. I'm a Software Developer with 5 years experience in technologies like: Java, Python, Spring, Hibernate, Redis, Gradle, Docker, Plus

$40 USD / heure
(0 Commentaires)
0.0
awsdojo2020

Have the experience of over 5 years in developing Big data pipelines over multiple technologies, hold certification in Hortonworks and Cloudera platforms. Currently employed full time as analytics application engineer Plus

$40 USD / heure
(0 Commentaires)
0.0
xcorp

Hi. I have a strong background in big data and Hadoop. I have worked as a Senior Data Platform Engineer and on technologies like spark, AWS, mapreduce, hbase, hive, Google bigquery, Kafka, Google cloud platform. I woul Plus

$55 USD / heure
(0 Commentaires)
0.0