Fermé

[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery

I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)

Deliverables

Complete writing cloud function which does the following:

1. Picks up data from pub/sub.

2. Processes the data and compares it to data in another table in BigQuery.

3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'

Please find project details in the attached excel file.

[login to view URL]

1. Data flow -> Pub/sub - Cloud function - Big Query

2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.

Compétences : Python, Architecture Logicielle, MySQL, Exploitation de Données, Hadoop

En voir plus : data entry processing officer, excel data entry processing controls, data entrydata processing, data entry processing skills, python script extract data web page, python script crape data, simply data email processing, python read input data file, scarborough data entry processing, python script extract data, python script read data text file, python script extract data website, expert data processing, python data stream processing, gcp pub sub ack, data being published to pub/sub must be sent as a bytestring., data being published to pub sub must be sent as a bytestring, google functions pub sub python

Concernant l'employeur :
( 56 commentaires ) Siem Reap, Cambodia

Nº du projet : #30956468

6 freelances font une offre moyenne de 71 $ pour ce travail

faizansaleem1598

Hi, I have a proven record for delivering quality websites to clients internationally, My reviews speaks for itself, I would like to offer you my services at a reasonable cost with a high end website for your business. Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(8 Commentaires)
4.9
dineshrajputit

hi, I can do this Hadoop, hive work since, I have expertise in bigdata technology with 9 years of experience. good command over Hadoop, hive, spark, nosql, java, Linux, aws, gcp... I am seeing similar job post from l Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(9 Commentaires)
3.9
DGTLServices

Hello there, I have looked through your details and I can certainly provide any services related to Python. I have 6 years of extensive experience in Python related projects as I have been working as a senior softwa Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(6 Commentaires)
3.5
MohsenKh94

Hello! This is Mohsen Khodadadi and I manage a team of software, frontend, backend, web developers, and graphic designers. I'd be happy to help you with your "[PYTHON & GCP EXPERT] Data ingestion and processing from Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(0 Commentaires)
0.0
MaryamRaad

I am Mraym, a computer engineer with more than 8 years of experience in python programming and database management. I have done numerous projects in database handling using python and other programs. Thus I can perform Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 2 jours
(0 Commentaires)
0.0
lucas121sacul

Hey, my current job involves doing something very similar to your requirements. IoT device -> pub/sub -> Topic -> SQL or Firestore (depending on the data). Given my dirrct experience with a similar project architecture Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 4 jours
(0 Commentaires)
0.0