Skip to main content

​
Overview

Google BigQuery is a fully managed, serverless data warehouse that enables scalable analysis over large datasets. Datazone connects to BigQuery using a Google Cloud service account, allowing you to extract tables and views from any dataset within your GCP project.

​
Connection Parameters

Set on the source when you create it. Names match the API and CLI payload.
​
string
required
Name — a unique identifier for your BigQuery source.
​
string
required
Project ID — your Google Cloud project ID, for example my-gcp-project. Query jobs are billed to this project.
​
string
required
Credentials JSON — the full contents of your service account key file, as JSON. Stored encrypted.

​
Extract Parameters

Set per extract, in source_parameters. See Shared Extract Parameters for mode, replication_key, and schema_mapping.
​
string
required
Table Name — the table or view to extract, qualified as dataset_id.table_id — for example analytics.events. The project comes from the source’s project_id, so do not include it here; a value that is not exactly two dot-separated parts is rejected.
​
integer
default:"1000000"
Rows fetched and written per batch.
​
integer
Hard cap on the total number of rows extracted, applied as a LIMIT on the generated query. Unset means no limit. Useful for sampling a large table before committing to a full extract.
​
boolean
default:"false"
Run the generated query with BigQuery legacy SQL instead of standard SQL. Leave off unless the table requires it.
​
string
default:"*"
Columns to read, as a comma-separated list. Selecting only what you need is the single biggest lever on BigQuery cost, since a columnar scan is billed by bytes read.
​
string
Row filter pushed down into the query, written without the WHERE keyword — for example status = 'active'. On a partitioned table, filtering on the partition column prunes partitions and lowers cost.

​
Setting Up a Service Account

  1. Go to IAM & Admin → Service Accounts in the Google Cloud Console
  2. Create a new service account (e.g. datazone-reader)
  3. Grant the following roles:
    • BigQuery Data Viewer — read access to datasets and tables
    • BigQuery Job User — permission to run query jobs
  4. Create a JSON key for the service account
  5. Copy the full contents of the downloaded JSON key file into the Credentials JSON field

​
Required Permissions

The service account needs the following IAM roles:
  • roles/bigquery.dataViewer — for reading table data
  • roles/bigquery.jobUser — for executing queries
For cross-project access, grant BigQuery Data Viewer on the specific datasets in the source project.

​
Limitations

  • BigQuery extracts use the BigQuery Storage API for efficient large-scale reads
  • Partitioned and clustered tables are supported
  • Supported BigQuery regions: all standard GCP regions

​
Next Steps

After configuring your BigQuery source:
  1. Create extracts to specify which tables or views to ingest
  2. Configure scheduling for recurring extracts
  3. Integrate the source into your data pipelines
For more information about working with extracts and pipelines, refer to their respective documentation sections.