Log inRequest a demoStart free

Learning center

Cross-system test-data subsetting

How to give QA one customer and every related policy, claim, account and transaction across several systems, without dangling references or unrelated records.

Tutorial

A typical test request sounds simple: "give QA this customer and everything needed to test the renewal workflow". In practice "everything" lives in several systems — the customer in a CRM, the policy in a policy administration database, claims in a claims system, payments in a warehouse. Cross-system Subsetting is the practice of selecting one coherent slice across all of them.

The slice has to be complete (every required related record is there), small (nothing unrelated), and repeatable (the same request yields the same records next week).

Building a referentially closed subsetStart from a driving entity selection, follow foreign keys up to required parents, include required children, close the reference set, then verify there are no dangling keys.Subsetting · Your environmentSelect drivingentitiesFollow keys toparentsInclude requiredchildrenClose thereference setVerify nodangling keys
Building a referentially closed subset. Start from a driving entity selection, follow foreign keys up to required parents, include required children, close the reference set, then verify there are no dangling keys.
Text description
  1. Subsetting (Your environment): Select driving entities → Follow keys to parents → Include required children → Close the reference set → Verify no dangling keys

Why it matters

Copying whole databases into test environments is slow, expensive and risky, so teams subset. But subsetting each system independently — "10% of customers here, 10% of policies there" — produces fragments that do not belong together. Tests then fail for reasons unrelated to the code under test: a policy whose customer is missing, a payment for a claim that was not selected.

The fix is to subset by Business entity rather than by table. Start from a root entity (a customer, a member, an account) and follow approved relationships outward until the Referential closure is reached: every record the root needs is included, and every record included belongs to a selected root.

Crossing systems adds two problems a single database does not have. Relationships between systems are not declared in any catalog, so they must be declared as mappings. And the traversal must stay bounded, or a customer with ten thousand transactions pulls a quarter of the warehouse into a unit-test dataset.

Referential integrity across tables and systemsParent records such as customers and accounts must exist for every child record such as transactions, cards and statements. When a key is masked, the same masked value must be used everywhere it appears so that joins still work across tables and systems.Parentscustomer (key C-1001)account (key A-2001 →C-1001)Childrentransaction → A-2001card → A-2001statement → A-2001Every child key resolves to a parent; masked keys stay consistent
Referential integrity across tables and systems. Parent records such as customers and accounts must exist for every child record such as transactions, cards and statements. When a key is masked, the same masked value must be used everywhere it appears so that joins still work across tables and systems.
Text description
  1. Parents: customer (key C-1001) → account (key A-2001 → C-1001)

    Connection: Every child key resolves to a parent; masked keys stay consistent

  2. Children: transaction → A-2001 → card → A-2001 → statement → A-2001

Example

A QA engineer asks for "three customers with an open claim, including their policies, claims, claim payments and the customer's accounts".

  1. Roots. The subsetter selects three customers from the CRM that match the filter (an open claim exists), using a fixed seed so the same request picks the same customers again.
  2. Within a system. From each customer it follows declared foreign keys in the CRM: addresses, contact preferences.
  3. Across systems. An approved relationship says crm.customer.customer_id corresponds to policy_admin.policy.holder_id. The subsetter selects those customers' policies, then follows the policy database's own keys to coverages and endorsements.
  4. Further hops. From policies to claims (claims system, via policy_number), from claims to payments (warehouse, via claim_id). A maximum depth and per-table row limits keep the traversal bounded.
  5. Parents for integrity. A payment references a payee bank record; that parent is included even though nobody asked for it, because leaving it out would create a dangling reference. Reference tables such as product codes are copied whole.
  6. Check. Before the dataset is certified, orphan detection confirms there are no dangling required references, and the manifest records which relationship brought in each table, without recording any values.

The result is a small, connected estate: three customers and only their records, in four systems, consistent with each other.

How DataNivra approaches it

The subsetter works on an entity graph built from three kinds of edges: foreign keys declared in each source, relationship templates supplied by industry packs (for example the insurance pack's policy–claim links), and explicit relationships you declare. Where the key value is shared between systems, the closure spans sources today: the engine's tests select roots in one source and pull the related rows from another, and certification refuses a dataset with orphaned rows.

Selection is deterministic for a given seed and policy, traversal depth and row limits are enforced, and the subset manifest records the provenance of every table — which relationship brought it in and which filters applied — as metadata only. Row-level work happens in the agent inside your network.

Where systems use different identifiers for the same entity (customer_id in one system, party_key in another), the subsetter needs an identity mapping between them. Owner-approved, versioned identity mappings are in active development, together with a synthetic multi-system estate that proves the rules: no dangling required relationships, no unrelated entity leakage, bounded traversal, deterministic selection and clear provenance.

Masking must be consistent across the same systems, or the carefully selected slice falls apart after masking: see cross-system deterministic masking. Try entity-aware subsetting on synthetic data in the interactive demo, estimate subset sizes with the free tools, and check which sources can be read today on the integrations page.

Key takeaways

  • Subset by business entity and follow relationships, not by percentage per table.
  • Declare cross-system relationships explicitly and keep traversal bounded.
  • A subset is not done until integrity is checked and provenance is recorded.