I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)
Complete writing cloud function which does the following:
1. Picks up data from pub/sub.
2. Processes the data and compares it to data in another table in BigQuery.
3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'
Please find project details in the attached excel file.
[login to view URL]
1. Data flow -> Pub/sub - Cloud function - Big Query
2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.
9 freelances font une offre moyenne de 175 $ pour ce travail
hi, I can do this Hadoop, hive work since, I have expertise in bigdata technology with 9 years of experience. good command over Hadoop, hive, spark, nosql, java, Linux, aws, gcp... please let me know if we can have q Plus
Hi, Hope you doing well, I have experience in python development and automation. I will be happy to help you. I'm confident i can exceed your expectations.
Hi! I'm an individual developer working in building websites and mobile apps. I have completed lots of projects in the field of programming websites, programming languages such as Wordpress, Laravel, Shopify, ... inclu Plus
Dear. Thanks for your job posting. I am glad to place a bid on here as I am a python expert and can help your project right now. I hope we will discuss in more detail about your project. Kind regards.
Hi, I understand your project requirements completely and feel confident to deliver the job perfectly. Kindly share some more details so I can start this job right away. I am expert programmer with expertise in PHP, W Plus
Hi, I'm a senior in Website and App Development. I read the job details extremely carefully. I think I can help you to complete your project with my rich knowledge and experience. I'm ready to do it now. Should you req Plus
I can understand about your project. I will solve you writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I am waiting.. Plus
Hi. Sir I can do the work with best quality within limited Time. I have 2 year experience with this type of work.