Guide
A test data management framework is the set of policies, processes and tooling that governs how test data is sourced, transformed, delivered and retired across an engineering organisation. Without one, teams make individual decisions — copying production databases, writing elaborate fixtures, or skipping tests for lack of safe data — and these decisions compound into real incidents: personal data in the wrong environment, a test that only passes because it shares state with another, or a QA cycle that stops because the database is stale.
A framework turns ad hoc decisions into a governed, repeatable workflow. This guide describes what a complete test data management framework covers and how to implement one.
Framework pillars
A mature test data management framework rests on four pillars.
Data governance. The framework establishes who can request test data, from which sources, under which policies, with which approval steps and for how long. Governance prevents the accidental spread of sensitive data and creates an audit trail. Without governance, the other pillars cannot be trusted: even excellent masking is meaningless if anyone can bypass it by requesting an unmasked dump.
Technical pipeline. The framework provides tooling that discovers sensitive columns, subsets and masks source data, generates synthetic records, certifies the result and delivers it to the test environment. The pipeline runs inside the organisation so raw production rows never leave the customer boundary. See data discovery and classification, data subsetting and data masking.
Environment coverage. Every test environment tier — developer workstations, shared QA environments, system integration testing (SIT), user acceptance testing (UAT) and performance environments — has different data volume and sensitivity requirements. The framework defines what each tier needs and how the pipeline satisfies it. See DEV vs QA vs SIT vs UAT vs Performance.
Measurement and freshness. The framework tracks which datasets were provisioned when, against which policy version, and schedules refresh so test environments do not drift from production schema changes. See refresh cadence.
The test data lifecycle
A framework governs the following lifecycle stages, in order.
Discovery and classification
Before any data can be safely moved or masked, every sensitive column must be identified. Discovery scans schema metadata and column profiles; classification assigns a sensitivity tier to each column. This is not a one-time step: schema drift adds new columns, and new source systems are added over time. A living classification registry is a prerequisite for every downstream stage. See data discovery and classification.
Subsetting
Copying a full production database to a test environment is the most common cause of data exposure incidents. Subsetting replaces the full copy with a small, consistent slice. A subset starts from a business entity — an account, a customer, an order — and follows foreign keys to every related record, across tables and across systems. The result is a fraction of the size of the original but behaves like the whole because no key dangles. See test data subsetting and referential integrity.
Masking and pseudonymization
Sensitive values in the subset are replaced with realistic, format-preserving substitutes. Masking is deterministic: the same source value always produces the same masked value, so a customer's pseudonym is consistent across the CRM, the data warehouse and the order management system. Masking runs inside the agent in the customer's own environment; the DataNivra control plane never receives a row. See data masking and the masking vs tokenization guide.
Synthetic data generation
Some test scenarios cannot be built from production samples: fraud patterns, edge-case account states, rare error conditions. Synthetic data generation creates realistic records from schema definitions and domain rules without any production input. The framework decides when synthetic data is appropriate and ensures synthetic records are clearly labelled as such, never mixed silently with masked production data. See synthetic vs masked production-like data.
Certification
Before a dataset is provisioned, certification checks that masking coverage is complete, referential integrity holds, the policy version matches the approved request, and the provenance chain is intact. A dataset that fails certification is not delivered. Certification creates a tamper-evident record that can answer the question "was this test environment masked?" at any future audit. See dataset certification.
Provisioning and refresh
The certified dataset is loaded into the target environment — a developer's local database, a shared QA schema, or a containerised test harness — with a defined expiry. When the underlying source data or schema changes, the dataset is refreshed by repeating the lifecycle from discovery rather than by patching rows in place. Refresh is typically automated and can be triggered from a CI/CD pipeline. See refresh cadence and TDM in CI/CD.
Governance model
A governance model defines the approval workflow and sets scope boundaries.
Request and approval. Developers, QA engineers and automation pipelines request datasets through a self-service interface or API. Each request cites a dataset definition — the source, entity, filters, masking policy — and goes through a configurable approval gate. High-sensitivity sources or large subsets may require a data steward approval; low-sensitivity synthetic bundles may be approved automatically. See the test data API docs and developers page.
Policy as code. Masking and classification policies are version-controlled. When a policy changes, existing certified datasets remain valid against the policy version they were built under; future requests pick up the new version. This makes the governance model auditable and reproducible.
Audit trail. Every provisioning event records the request, the approver, the policy version, the masking coverage and the expiry. This trail supports internal audit, data-breach investigations and regulatory enquiries without requiring manual records.
Integration with CI/CD and developer workflows
A framework that requires a ticket and a one-week wait will be bypassed. To succeed, the framework must deliver data fast enough that the alternative — copying production — is not attractive.
Modern pipelines integrate the test data lifecycle with CI/CD: a pull request triggers a dataset provisioning event; the test suite runs against the fresh data; the dataset expires when the branch is deleted. DataNivra supports this workflow through its Test Data API, TypeScript and Python SDKs, and a CLI that runs inside container images.
See all available integrations for source and target system support, and the CI/CD use cases for concrete examples.
Where to start
If you are building or maturing a test data management framework, the TDM readiness assessment is a structured way to evaluate your current state against the framework's pillars. The assessment scores governance, pipeline maturity, environment coverage and measurement, and suggests the highest-value improvements.
When you are ready to implement, the documentation hub has API references, SDK guides and agent installation instructions. To try synthetic data without an agent, visit the free tools or open the Start free page to create an account.