Integrations · Streaming · Available now

Apache Kafka test data, inside your network

Kafka connector: declared topics read as a bounded snapshot window (log start or a start time up to the last stable offset when the job starts), with JSON fields, text or binary values, record metadata columns and offset-range row counts. It never joins a consumer group and never commits offsets.

Available now Streaming

Shipped in the standard agent and backed by recorded conformance evidence.

What works today

Kafka connector: declared topics read as a bounded snapshot window (log start or a start time up to the last stable offset when the job starts), with JSON fields, text or binary values, record metadata columns and offset-range row counts. It never joins a consumer group and never commits offsets.

SASL/SCRAM (or SASL/PLAIN) over TLS with the broker certificate verified; plaintext only on loopback or by explicit opt-in. Consumers use explicit partition assignment with read_committed isolation. The driver (confluent-kafka) ships in the standard agent image.

Authentication: SASL/SCRAM-SHA-512 or SCRAM-SHA-256 (secret reference); SASL/PLAIN over TLS.

Known limitations

  • Requires agent 0.4.0 or later; earlier agent images do not include the confluent-kafka driver.
  • Reads a bounded snapshot window per job; it never follows a live stream.
  • Row counts are offset ranges: exact for ordinary records, an upper bound when compaction or transaction markers occupy offsets.
  • Topics declare no keys: subsetting and relationships come from industry-pack templates or declared relationships and are closed only within the snapshot window.
  • JSON field shape is inferred from a bounded sample or declared; Avro and Protobuf with a Schema Registry are not decoded (read such topics as binary).
  • SASL authentication only; mutual-TLS client certificates are not supported yet.
  • Masked output goes to the configured provisioning target; nothing is produced back to Kafka.

Capability matrix

Each capability is tracked separately and shown at the level the recorded conformance evidence supports. A connector that only discovers metadata is never presented as equivalent to one that runs the whole workflow.

Capabilities of Apache Kafka
CapabilityLevel
DiscoverySupported
ClassificationSupported
ProfilingSupported
MaskingSupported
Deterministic maskingSupported
Entity-aware subsettingSupported
Relationship preservationSupported
Synthetic workflowsSupported
CertificationSupported
ProvisioningSupported
CI/CDSupported
Evidence generationSupported

Read-only proof

Proven before reading: the brokers' authorized operations (KIP-430) on every declared topic and on the cluster must include none of WRITE, CREATE, DELETE, ALTER, ALTER_CONFIGS, IDEMPOTENT_WRITE, CLUSTER_ACTION or ALL. A broker that returns no authorized operations, or runs without an authorizer, is refused.

Conformance evidence

DataNivra runs every connector through a common conformance kit on synthetic data: no write statements, proven read-only access, secret redaction, schema and relationship discovery, bounded streaming, cancellation, clear reason codes, egress canaries and no raw values in logs, then the discover, mask, subset, certify and provision workflow. The published status is computed from those results.

Environments whose conformance run met the preview minimum
Engine or serviceVersionEnvironmentDriverAuthenticationVerified on
Apache Kafka4.1.0real-engineconfluent-kafka 2.16.0 (librdkafka 2.16.0)SASL/SCRAM-SHA-512 over TLS (test CA verified)2026-10-08

Connector version 0.1.0, last verified 2026-10-08.

What crosses the boundary

Connectors run inside the agent in your network. Only metadata, aggregates and evidence reach DataNivra Cloud:

  • Schema, table and column names and data types (for file sets, operator-declared or opaque table names — never data-derived file names).
  • Aggregates such as row-count estimates, null and distinct ratios; counts from 1 to 10 are reported as 10 (small-cell protection).
  • Classification findings (which column looks like which kind of sensitive data) and relationship metadata.
  • Evidence metadata: checksums, gate outcomes, engine and policy versions, and templated error codes.

Never sent:

  • Any cell value, row, sample, minimum/maximum or top-k value — masked or not.
  • Credentials, connection strings or resolved secret values: DataNivra stores only the secret reference.
  • Free text copied from your data, or exception messages that contain values.

See the customer-resident architecture and how to verify these claims yourself.

Setup outline

  1. Use agent 0.4.0 or later: the standard agent image includes the confluent-kafka driver.
  2. Create a dedicated SASL/SCRAM user with READ and DESCRIBE ACLs on the topics to read and DESCRIBE on the cluster, and nothing else (no WRITE, CREATE, DELETE, ALTER or consumer-group ACLs).
  3. Store the connection document with the password as a secret reference. SASL_SSL with certificate and host-name verification is the default; declare each topic and how its values decode (json, text or binary).
  4. In the console, register the source with the secret reference of that document (for example vault://tdm/source-connection) and the agent that can reach it, then run discovery.
  5. Before reading, the agent asks the brokers which operations the user is authorized for and refuses to continue if any could write. Each job reads a bounded snapshot window and never commits offsets.

Example connection document, stored in your secret store and resolved by the agent locally (DataNivra only ever sees its reference):

{
  "kind": "KAFKA",
  "bootstrap_servers": ["kafka-1.example.internal:9093"],
  "topics": { "members": {}, "claims": {} },
  "username": "tdm_reader",
  "password_ref": "vault://tdm/kafka-tdm-reader#password"
}

Next: install the agent, then follow the getting started guide.

All integrations · Streaming test data management · Guides · Start free