Integrating Google BigQuery usage with Dagster+ Insights#

External metrics, such as Google BigQuery usage, can be integrated into the Dagster Insights UI. The dagster-cloud package contains utilities for capturing and submitting external metrics about data operations to Dagster+ via an API.

If you use the BigQuery resource, use this guide to integrate BigQuery metrics into the Insights UI. For instructions on integrating usage of dbt models that run in BigQuery, see Integrating BigQuery & dbt with Dagster+ Insights.

Prerequisites#

To complete the steps in this guide, you'll need:

• A Dagster+ account on the Pro plan

• Access to the Dagster+ Insights feature

• BigQuery credentials that have access to the INFORMATION_SCHEMA.JOBS table. For more information on granting access to this table, see the BigQuery documentation.

• To install the following libraries:

  pip install dagster dagster-cloud dagster-gcp
  

  Note: If you already have dagster-cloud installed, make sure you're using version 1.7.0 or newer.

Step 1: Replace your BigQuery resources#

The first step is to replace any existing BigQuery resources with InsightsBigQueryResource. This resource is a drop-in replacement for the BigQueryResource resource, but it also emits BigQuery usage metrics to the Dagster+ Insights API.

from dagster_cloud.dagster_insights import InsightsBigQueryResource

defs = Definitions(
  resources={
    "bigquery": InsightsBigQueryResource(project="my-project")
  }
)

Step 2: View BigQuery usage in the Dagster UI#

Once the pipeline runs, BigQuery usage will be visible in the Insights tab in the Dagster UI:

The BigQuery cost metric is based off of the bytes billed for queries performed using the InsightsBigQueryResource, based on a unit price of $6.25 USD per TiB.

Related#