Existing Flow:

Iot Devices(Sending Json Payload) -> Kafka Streaming(receiving & Producing) -> Spark Streaming(Consuming Json using Scala) -> Cassandra(For Storage)

Project Description:

IoT devices getting uploaded with JSON payload in Cassandra.

IoT devices sending device information through a simulator to Kafka streaming.

Kafka Producer producing JSON streaming data.

Kafka JSON data-consuming in Scala 2.11.11 with Spark 2.4.7

Spark consumed data getting inserted into Cassandra.


1. Performance Tuning

-> Need to persist @100/sec JSON payload into Cassandra using spark streaming

2. Project should support higher version of Cassandra Spark Connector with latest Spark and Scala

(Need to validate code on higher version compatibility)

Note: -> We've already achieved @40/sec. We'll share it.

-> Project deployed on Linux( RAM 16 GB, 8 CPU Cores)

-> We tested on Spark 2.4.7, Scala 2.11.11, and Spark Cassandra Connector 2.4.0

As per the required skillset and interest will share the code.

Compétences : Scala, Spark, Cassandra, Apache Maven

En voir plus : scala redis cassandra, spark cassandra connector, spark cassandra, add functionality to existing website, adding feedback section to existing website, changes to existing php website, add shopping cart to existing website, alterations to existing page contest freelancer, integrate configurator plugin to existing woocommerce website. -- 2, Modifications to existing website, how to connect iot devices to internet, spark cassandra connector scala example, which two devices are used to connect iot devices to a home network, which two devices are used to connect iot devices to a home network?, web-based attacks to discover and control local iot devices, how to secure iot devices, spark write to cassandra example, how to control iot devices, role of an operating system to support software in iot devices, two authentication protocols specified to iot devices

Concernant l'employeur :
( 0 commentaires ) Ghaziabad, India

Nº du projet : #30953996

3 freelances font une offre moyenne de 26778 ₹ pour ce travail


hi, I can do this Hadoop, hive work since, I have expertise in bigdata technology with 9 years of experience. good command over Hadoop, hive, spark, streaming, Kafka, nosql, java, scala, Linux, aws, gcp... please let Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% INR en 20 jours
(3 Commentaires)

Hello, Cassandra needs full server installation for best results, so better use an independent cluster for purpose. secondly if you are keen on using json persristance use mongodb/dynamodb instead of cassandra, which h Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% INR en 7 jours
(0 Commentaires)

Enthusiastic professional with 4+ years of work experience in Data Science, Machine Learning, Natural language processing, Deep Learning pipelines, Data analytics, Software development and Big Data processing as an Azu Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% INR en 7 jours
(0 Commentaires)