Google BigQuery test data, inside your network

What works today

BigQuery connector: dataset and table discovery through the tables API, partition and clustering metadata, row counts from table metadata, and reads through tabledata.list, which is not billed as query scanning.

Service-account key (limited to the bigquery.readonly scope) or workload identity. Views need query jobs, and every query job sets maximum_bytes_billed. The driver ships in the standard agent image from agent 0.3.0.

Authentication: service-account key (secret reference); workload identity / application default.

Known limitations

• Verified against a real Google Cloud project with service-account keys. A key is limited to the read-only scope, which cannot run query jobs: tables are read and counted without jobs, but views need application-default credentials (for example the agent's attached service account).
• Requires agent 0.3.0 or later; earlier agent images do not include its driver.
• BigQuery primary and foreign keys are unenforced; relationships may need explicit mappings.
• Profiles are computed by the agent over a bounded sample of listed rows.

Capability matrix

Each capability is tracked separately and shown at the level the recorded conformance evidence supports. A connector that only discovers metadata is never presented as equivalent to one that runs the whole workflow.

Read-only proof

Proven before reading: testIamPermissions on every exposed table must grant none of updateData, update, delete, setIamPolicy or setCategory; if the permission API cannot answer, the source is refused.

Conformance evidence

DataNivra runs every connector through a common conformance kit on synthetic data: no write statements, proven read-only access, secret redaction, schema and relationship discovery, bounded streaming, cancellation, clear reason codes, egress canaries and no raw values in logs, then the discover, mask, subset, certify and provision workflow. The published status is computed from those results.

Connector version 0.1.0, last verified 2026-10-07.

What crosses the boundary

Connectors run inside the agent in your network. Only metadata, aggregates and evidence reach DataNivra Cloud:

• Schema, table and column names and data types (for file sets, operator-declared or opaque table names — never data-derived file names).
• Aggregates such as row-count estimates, null and distinct ratios; counts from 1 to 10 are reported as 10 (small-cell protection).
• Classification findings (which column looks like which kind of sensitive data) and relationship metadata.
• Evidence metadata: checksums, gate outcomes, engine and policy versions, and templated error codes.

Never sent:

• Any cell value, row, sample, minimum/maximum or top-k value — masked or not.
• Credentials, connection strings or resolved secret values: DataNivra stores only the secret reference.
• Free text copied from your data, or exception messages that contain values.

See the customer-resident architecture and how to verify these claims yourself.

Setup outline

1. Use agent 0.3.0 or later (earlier agent images do not include the BigQuery driver).
2. Create a service account with BigQuery Data Viewer on the dataset and Job User on the project (no Data Editor, Data Owner or Editor).
3. Store the connection document with the service-account key as a secret reference. Tables are read without query jobs; views need application-default credentials.
4. In the console, register the source with the secret reference of that document (for example vault://tdm/source-connection) and the agent that can reach it, then run discovery.
5. Before reading, the agent checks that no exposed table grants write permissions and refuses to continue if one does.

Example connection document, stored in your secret store and resolved by the agent locally (DataNivra only ever sees its reference):

{
  "kind": "BIGQUERY",
  "project": "my-project",
  "datasets": ["claims"],
  "credentials_json_ref": "vault://tdm/bigquery-tdm-agent#key_json"
}

Next: install the agent, then follow the getting started guide.