Random Isn't Realistic: Why Test Data Needs to Know About Your Domain

Most fixture generators fill a schema field by field: a random string here, a random int there, a random date somewhere in the last year. Each value, in isolation, looks fine. Run the test suite and it's green. Ship to staging with real-shaped data and it falls over immediately. The gap isn't in any single field — it's in the relationships between fields that independent random generation never bothered to model, which is exactly what referential integrity test data generation is supposed to fix.

The bug is never in one field

A status enum with a valid value, a created_at that's a real timestamp, an amount that's a plausible number — none of these will ever fail a naive test, because naive tests check the same thing the generator produces: is each field individually well-formed? The actual logic bugs live one level up, in constraints that span two or more fields, or two or more records. If your generator doesn't know those constraints exist, it can't violate them — but it also can't exercise them, which means your tests never see the code path that handles them.

Referential integrity test data generation: order matters, literally

Take an order record with created_at and shipped_at. Business logic almost certainly assumes shipped_at >= created_at — a refund calculation, an SLA report, a "days to ship" metric. A generator that picks both timestamps independently and uniformly at random will, roughly half the time, produce an order that shipped before it was created. Two things can happen next, and both are bad. Either the test fails on a technicality unrelated to what you're actually testing (so someone "fixes" it by clamping the dates or removing the assertion), or the field is nullable and the generator just leaves shipped_at empty every time, which passes trivially and never tests the shipped-order path at all.

The same pattern shows up with discount_code foreign keys. A code generated as a random string will essentially never match a real row in your coupons table. Tests either don't check the join (so a broken discount lookup ships unnoticed) or the field is nulled out to dodge the issue (so the "order with a valid discount applied" path — arguably the one most worth testing — never runs). Correct-by-construction fixtures fix this at generation time: shipped_at is derived as created_at + a delay, and discount_code is picked from the set of codes you actually generated, so the invariant holds by design and the interesting cases are the default, not an accident.

Real data isn't uniform

Even when a generator respects an invariant, it usually gets the shape of the data wrong. Real-world distributions are lopsided. Most users place zero or one order; a small number place hundreds. Most support tickets get closed same-day; a long tail sits open for weeks. Most timestamps cluster during business hours in a handful of timezones, not evenly across all 24.

Uniform random generation misses both ends of that distribution at once. It doesn't produce enough of the overwhelmingly common case — the zero-order user, the same-day ticket — to make sure your code handles it cheaply and correctly. And it doesn't produce the genuine long tail either, because "pick a random integer between 0 and 100" doesn't generate the same stress patterns as "pick from a power-law distribution where a few accounts have 500 orders." Pagination bugs, N+1 queries, and UI layouts that break under a long list all hide in that tail, and uniform randomness will happily generate a thousand records without ever producing it.

Cross-record consistency

The relationships that matter aren't always between two fields on one record — sometimes they're between records, or between a record and its children. An invoice's stored total should equal the sum of its line items. A user's country, currency, and phone_number format should agree with each other. If you generate each field independently, per record, per table, these correlations vanish, and nothing forces them back.

That's usually fine for a "does the page render" smoke test. It's not fine for testing the code that actually computes an invoice total from line items, or that formats a phone number based on locale — because the fixture already has the answer baked in as a mismatch, and no assertion will catch a bug that produces an equally wrong, equally inconsistent result. Worse, some of these bugs are silent in production, too: a currency mismatch just displays a slightly wrong number until someone notices during an audit. Catching it requires test data where the correlation exists in the first place, so the code has to actually get it right to pass.

Catching this earlier, cheaper

None of these are new bugs — they're the kind that eventually surface in staging, or in a support ticket, or in a very awkward audit. The expensive part is when you find them. A bug caught by a unit test that generates a domain-consistent order-with-discount fixture costs a few seconds of CI time. The same bug caught in production costs an incident, a hotfix, and possibly a customer.

This is the practical case for structured, template-driven generation over field-by-field randomness: a template can encode "derive shipped_at from created_at," "pick discount_code from the generated coupon set," "sample order count from a power-law distribution," and "keep country and currency in sync" as explicit rules, not hopeful accidents. That's the difference between test data that happens to satisfy your schema and test data that actually resembles the data your system will have to survive — and it's the gap JSON Fabrica's generation model is built to close, by letting relationships and distributions be first-class parts of the template instead of afterthoughts bolted on with a validation pass.

FAQ

Why does randomly generated test data still cause bugs that a green test suite missed? Naive random generation only checks that each field is individually well-formed, not that relationships between fields or records hold. A shipped_at picked independently of created_at can end up before it, and a discount_code generated as a random string will never match a real coupon row, so the code paths that depend on those relationships never actually run in tests.

What is referential integrity in test data generation? It means generated records respect the same cross-field and cross-record rules real data has to follow — an order's shipped_at coming after its created_at, or a discount_code pointing at a coupon that actually exists in the generated batch — instead of each field being picked independently at random.

How do I generate test data with realistic, non-uniform distributions? Sample from a distribution that matches your domain instead of a uniform random range, for example a power-law distribution where a small number of accounts hold most of the order volume. Uniform randomness misses both the common zero-order case and the genuine long tail that pagination and N+1 query bugs tend to hide in.

How do I keep related records like customers and orders consistent when generating test data? Generate related documents in the same call and wire one field to another's value, for example an order's customerId pointing at a customer generated in the same batch, rather than generating each table independently and hoping the IDs line up. JsonFabrica's batch generation API supports this directly through a relations map with a round-robin strategy.