Fermé

[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 3

I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)

Deliverables

Complete writing cloud function which does the following:

1. Picks up data from pub/sub.

2. Processes the data and compares it to data in another table in BigQuery.

3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'

Please find project details in the attached excel file.

[login to view URL]

1. Data flow -> Pub/sub - Cloud function - Big Query

2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.

Compétences : Python, Architecture Logicielle, MySQL, Exploitation de Données, Hadoop

En voir plus : data entry processing officer, excel data entry processing controls, data entrydata processing, data entry processing skills, python script extract data web page, python script crape data, simply data email processing, python read input data file, scarborough data entry processing, python script extract data, python script read data text file, python script extract data website, expert data processing, python data stream processing, gcp pub sub ack, data being published to pub/sub must be sent as a bytestring., data being published to pub sub must be sent as a bytestring, google functions pub sub python

Concernant l'employeur :
( 56 commentaires ) Siem Reap, Cambodia

Nº du projet : #30970401

9 freelances font une offre moyenne de 175 $ pour ce travail

dineshrajputit

hi, I can do this Hadoop, hive work since, I have expertise in bigdata technology with 9 years of experience. good command over Hadoop, hive, spark, nosql, java, Linux, aws, gcp... please let me know if we can have q Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 15 jours
(9 Commentaires)
3.9
(4 Commentaires)
3.0
mohannetwork24

Hi, Hope you doing well, I have experience in python development and automation. I will be happy to help you. I'm confident i can exceed your expectations.

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(5 Commentaires)
2.2
artocenter

Hi! I'm an individual developer working in building websites and mobile apps. I have completed lots of projects in the field of programming websites, programming languages such as Wordpress, Laravel, Shopify, ... inclu Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(3 Commentaires)
2.4
MaksimKholin

Dear. Thanks for your job posting. I am glad to place a bid on here as I am a python expert and can help your project right now. I hope we will discuss in more detail about your project. Kind regards.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(2 Commentaires)
1.0
smhuda

Hi, I understand your project requirements completely and feel confident to deliver the job perfectly. Kindly share some more details so I can start this job right away. I am expert programmer with expertise in PHP, W Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(1 Évaluation)
0.6
tesev95711

Hi, I'm a senior in Website and App Development. I read the job details extremely carefully. I think I can help you to complete your project with my rich knowledge and experience. I'm ready to do it now. Should you req Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(0 Commentaires)
0.0
sumyalmm

I can understand about your project. I will solve you writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I am waiting.. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(0 Commentaires)
1.8
madananp810

Hi. Sir I can do the work with best quality within limited Time. I have 2 year experience with this type of work.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(0 Commentaires)
0.0