Deeper into Dataform 3: Auditing Dataform

It's important to monitor Dataform - jobs executed by Dataform can be the primary source of BigQuery costs in a modern data platform. Forgetting to incrementalise a table, using a table instead of a view in the wrong place or performing complex window functions on a large table can all incur large costs and long run times. Using the WorkflowInvocationAction for each job we can extract its BigQuery Job ID, then extract key metadata for each BigQuery job by querying INFORMATION_SCHEMA.JOBS_BY_PROJECT , before writing the output back to BigQuery so that it can be analysed (maybe even by transforming it in Dataform). from google.cloud import dataform_v1 from google.cloud import bigquery from datetime import datetime # ------------------------------------------------------------ # CONFIG # ------------------------------------------------------------ PROJECT_ID = " my-project " REGION = " europe-west2 " REPOSITORY_ID = " analytics " WORKFLOW_INVOCATION_ID = " 123456789 " BQ_REGION = " region-europe-west2 " OUTPUT_TABLE = " my-project.raw_dataform_monitoring.raw_dataform_bigquery_metrics " # ------------------------------------------------------------ # CLIENTS # ------------------------------------------------------------ dataform = dataform_v1 . DataformClient () bq = bigquery . Client ( project = PROJECT_ID ) repository = dataform . repository_path ( PROJECT_ID , REGION , REPOSITORY_ID ) invocation_name = f " { repository } /workflowInvocations/ { WORKFLOW_INVOCATION_ID } " # ------------------------------------------------------------ # 1. GET WORKFLOW INVOCATION ACTIONS → EXTRACT JOB IDS # ------------------------------------------------------------ job_ids = set () actions = dataform . list_workflow_invocation_actions ( parent = invocation_name ) for action in actions : # only BigQuery actions contain job metadata if hasattr ( action , " bigquery_action " ) and action . bigquery_action : if action . bigquery_action . job_id : job_ids . add ( action . bigquery_action