Databricks test data, inside your network

What works today

Databricks connector: Unity Catalog catalog.schema.table discovery of Delta tables and views through a Databricks SQL warehouse, informational keys and bounded Arrow reads.

OAuth machine-to-machine (service principal) or a personal access token, both as secret references. Queries run on the SQL warehouse you configure under a statement timeout. The driver ships in the standard agent image from agent 0.3.0.

Authentication: OAuth M2M service principal (secret reference); personal access token (secret reference).

Known limitations

• Verified against a real workspace (Unity Catalog, serverless SQL warehouse) with OAuth machine-to-machine; personal access tokens are accepted but not part of the recorded run.
• Requires agent 0.3.0 or later; earlier agent images do not include its driver.
• Unity Catalog does not enforce foreign keys; informational constraints are read where declared.
• Workspace and metastore administrator rights are invisible to the read-only proof: use a dedicated non-admin service principal.
• Tags are read locally only; external locations, volumes, Delta Sharing and Jobs are not supported.

Capability matrix

Each capability is tracked separately and shown at the level the recorded conformance evidence supports. A connector that only discovers metadata is never presented as equivalent to one that runs the whole workflow.

Read-only proof

Proven before reading: the catalog, schema and table privileges of the service principal and its groups may include only USE CATALOG, USE SCHEMA, SELECT and BROWSE, and it must own no catalog, schema or table; anything else refuses the source.

Conformance evidence

DataNivra runs every connector through a common conformance kit on synthetic data: no write statements, proven read-only access, secret redaction, schema and relationship discovery, bounded streaming, cancellation, clear reason codes, egress canaries and no raw values in logs, then the discover, mask, subset, certify and provision workflow. The published status is computed from those results.

Connector version 0.1.0, last verified 2026-10-07.

What crosses the boundary

Connectors run inside the agent in your network. Only metadata, aggregates and evidence reach DataNivra Cloud:

• Schema, table and column names and data types (for file sets, operator-declared or opaque table names — never data-derived file names).
• Aggregates such as row-count estimates, null and distinct ratios; counts from 1 to 10 are reported as 10 (small-cell protection).
• Classification findings (which column looks like which kind of sensitive data) and relationship metadata.
• Evidence metadata: checksums, gate outcomes, engine and policy versions, and templated error codes.

Never sent:

• Any cell value, row, sample, minimum/maximum or top-k value — masked or not.
• Credentials, connection strings or resolved secret values: DataNivra stores only the secret reference.
• Free text copied from your data, or exception messages that contain values.

See the customer-resident architecture and how to verify these claims yourself.

Setup outline

1. Use agent 0.3.0 or later (earlier agent images do not include the Databricks driver).
2. Create a dedicated, non-admin service principal with an OAuth secret; grant it USE CATALOG, USE SCHEMA and SELECT, and Can use on one SQL warehouse.
3. Store the connection document with the client secret as a secret reference.
4. In the console, register the source with the secret reference of that document (for example vault://tdm/source-connection) and the agent that can reach it, then run discovery.
5. Before reading, the agent proves the principal holds only read privileges and owns no catalog, schema or table, and refuses to continue if it does.

Example connection document, stored in your secret store and resolved by the agent locally (DataNivra only ever sees its reference):

{
  "kind": "DATABRICKS",
  "server_hostname": "dbc-0000000-0000.cloud.databricks.com",
  "http_path": "/sql/1.0/warehouses/0123456789abcdef",
  "catalog": "claims",
  "schemas": ["claims"],
  "client_id": "00000000-0000-0000-0000-000000000000",
  "client_secret_ref": "vault://tdm/databricks-tdm-agent#client_secret"
}

Next: install the agent, then follow the getting started guide.

Using DataNivra with Databricks

The Databricks connector is available now: it ships in the standard agent and passed every conformance check.

Unity Catalog discovery Supported

Discovery walks the three-level catalog.schema.table namespace that the service principal can browse, and records column names, types, informational primary and foreign key constraints, table comments and tags as metadata. Tags such as a sensitivity label are useful classification hints; they are read, never written back.

Delta tables through a SQL warehouse Supported

Reads are ordinary SELECT statements executed on the Databricks SQL warehouse you configure, fetched as Arrow batches by the agent inside your network. Delta time travel is not used to read history: the connector reads the current version of each table, so older versions of sensitive rows are never pulled into a test dataset.

OAuth service principal and secret references Available now

The design authenticates as a Databricks service principal with OAuth machine-to-machine credentials (or a personal access token), both supplied as secret references that the agent resolves locally. DataNivra Cloud stores only the reference, never the secret.

Proven read-only access Available now

Before reading, the connector checks the grants of its principal: only USE CATALOG, USE SCHEMA, SELECT and BROWSE are acceptable. MODIFY, CREATE or ALL PRIVILEGES stops the job, so a mis-scoped principal fails closed instead of reading with write rights.

Bounded, cost-conscious reads Supported

Discovery reads metadata only. Profiles and subset reads run queries on your warehouse, bounded by row limits and a statement timeout, so a test-data job cannot turn into an unbounded scan of a production lakehouse.

Deterministic masking across systems Supported

Masking runs in the agent with keys that stay in your environment. Deterministic, keyed pseudonymisation gives the same customer the same masked key in Databricks and in the operational databases it came from, so joins between a lakehouse table and a PostgreSQL or SQL Server table still work in test.

Entity-aware subsets Supported

Unity Catalog does not enforce foreign keys, so relationships come from informational constraints where declared and from explicit relationship mappings you approve. The subsetter then follows them to pull a customer with the claims, policies and events that belong to them, and nothing unrelated.

Certification before provisioning Supported

A masking job that finished is not a certified dataset. Certification checks masking coverage, relationship integrity, policy versions, row counts and the manifest, and provisioning refuses any dataset that failed or was revoked.

CI/CD and evidence Supported

Pipelines request certified datasets through the REST API, and each dataset carries an evidence record — policy and engine versions, gate results and checksums, never row values — that an auditor can review later.

Not in scope for the first version: external locations, volumes, Delta Sharing and Databricks Jobs. They will be added only after they are tested.