I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)
Complete writing cloud function which does the following:
1. Picks up data from pub/sub.
2. Processes the data and compares it to data in another table in BigQuery.
3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'
Please find project details in the attached excel file.
[login to view URL]
1. Data flow -> Pub/sub - Cloud function - Big Query
2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.
6 freelances font une offre moyenne de 71 $ pour ce travail
Hi, I have a proven record for delivering quality websites to clients internationally, My reviews speaks for itself, I would like to offer you my services at a reasonable cost with a high end website for your business. Plus
hi, I can do this Hadoop, hive work since, I have expertise in bigdata technology with 9 years of experience. good command over Hadoop, hive, spark, nosql, java, Linux, aws, gcp... I am seeing similar job post from l Plus
Hello there, I have looked through your details and I can certainly provide any services related to Python. I have 6 years of extensive experience in Python related projects as I have been working as a senior softwa Plus
Hello! This is Mohsen Khodadadi and I manage a team of software, frontend, backend, web developers, and graphic designers. I'd be happy to help you with your "[PYTHON & GCP EXPERT] Data ingestion and processing from Plus
I am Mraym, a computer engineer with more than 8 years of experience in python programming and database management. I have done numerous projects in database handling using python and other programs. Thus I can perform Plus
Hey, my current job involves doing something very similar to your requirements. IoT device -> pub/sub -> Topic -> SQL or Firestore (depending on the data). Given my dirrct experience with a similar project architecture Plus