Deterministic Test Data: Replaying a Flaky Test Failure From Its Seed

Deterministic test data is the difference between debugging a flaky test and shrugging at it. A test that fails about one run in thirty on a random fixture is nearly un-debuggable: by the time you open the CI log the triggering data is already gone, you re-run the job, it comes back green, and nobody learns anything. The fix is to make your generated data reproducible — same inputs, byte-identical output — so a failure you saw once can be replayed on demand instead of waited for. This post is about that specific debugging workflow: capture the seed, then replay it.

What makes deterministic test data reproducible?

JsonFabrica generation is a pure function of (template, seed, params). Give it the same template, the same seed, and the same params and it returns the same output — every field, every record, byte for byte. There is no hidden entropy in the generator itself; the randomness is entirely driven by the seed. That is what "deterministic" buys you, and reproducibility is the payoff: a dataset is fully described by those three inputs, so you can regenerate it exactly whenever you hold the seed.

The seed is the handle. When you call POST /v1/templates/{templateId}/generate and supply a seed, that seed drives the whole result:

curl -s -X POST https://api.jsonfabrica.com/v1/templates/tpl_9f3a1c2e/generate \
  -H 'Authorization: Bearer sk_...' \
  -H 'content-type: application/json' \
  -d '{ "seed": 123456789, "params": { "name": "Jane Doe" } }'

{
  "data": "Hello Jane Doe, order #1001",
  "meta": { "seed": 123456789, "templateId": "tpl_9f3a1c2e", "generatedAt": "..." }
}

Run that request again with the same body and data is identical. The seed you passed comes back in meta.seed — and, crucially, so does a seed you didn't pass.

Why do random test fixtures cause flaky tests?

Field-by-field random data is great at coverage and terrible at reproducibility. Each run picks fresh values, so eventually some combination — a name with an apostrophe, a date on a leap day, a number that rounds the wrong way — trips a bug your code never handled. That is the random fixture doing its job: surfacing a real edge case. The problem is what happens next. The values that caused the failure existed only in that run's memory; the next run rolls new ones, the test passes, and the signal evaporates. You are left with a red build you cannot reproduce and a green re-run that proves nothing.

This is the flip side of a benefit covered in Templates vs. AI Synthetic Data: template-based generation is deterministic and controllable in a way free-form AI output is not, and that determinism is exactly what makes a failing run recoverable. Random values are fine — random and unrecoverable is the trap.

How do I reproduce a test failure caused by random data?

Capture the seed on every run, then replay it. The engine cooperates: when you omit seed, the service generates one (an auto-seed) and always echoes the seed it actually used back in meta.seed. That echoed value is your replay handle. So you never have to choose between fresh random data and reproducibility — you get both, as long as you log the seed.

The pattern is a thin wrapper that records meta.seed on every call:

async function generate(templateId, body = {}) {
  const res = await fetch(
    `https://api.jsonfabrica.com/v1/templates/${templateId}/generate`,
    {
      method: "POST",
      headers: { authorization: `Bearer ${process.env.JF_KEY}`,
        "content-type": "application/json" },
      // Pin the seed if the caller supplied one; otherwise let the service pick.
      body: JSON.stringify({ seed: process.env.SEED, ...body }),
    },
  );
  const json = await res.json();
  console.log(`[jsonfabrica] template=${templateId} seed=${json.meta.seed}`);
  return json.data;
}

Every run now prints a line like [jsonfabrica] template=tpl_9f3a1c2e seed=884211307. When a run goes red, the failing seed is right there in the CI log. Reproduce it locally with a one-line convention:

SEED=884211307 npm test

Because SEED flows into the request body, the wrapper pins generation to that seed and you get the identical failing dataset on your machine — first try, no waiting for the one-in-thirty roll to come up again. This is the same capture-the-seed discipline that makes browser runs reproducible in Playwright End-to-End Test Data; here it is aimed squarely at debugging.

Should I use a fixed seed or a random seed in CI?

Both — the trick is knowing where each belongs. The default should be capture, not pin: let each CI run pick a fresh seed so you keep surfacing new bad combinations, and log meta.seed so any failure stays recoverable. Pinning everything would freeze your coverage at whatever one seed happens to exercise.

Then pin deliberately in the places where you want stability rather than discovery:

• PR gates, visual-regression, and snapshot tests pass an explicit seed. These compare against a baseline, so the data must not move between runs or every diff is noise.
• A nightly fuzz job omits the seed entirely. Its whole purpose is to roll as many combinations as possible overnight and catch the rare ones, logging each seed so a hit can be pinned tomorrow.

That way your fast feedback loop is stable and your slow loop is adventurous, and a discovery from the fuzz job graduates into a pinned regression test the moment you copy its seed.

Reproducing a whole batch, not just one record

Most real fixtures are relational — customers with orders, accounts with transactions — and batches are deterministic too. POST /v1/batches takes a single seed on the BatchSpec, and every document is derived from that seed plus its index, so one seed reproduces the entire relational set:

{
  "seed": 20260908,
  "sequenceNamespace": "replay-884211307",
  "variableNamespace": "replay-884211307",
  "documents": [
    { "templateId": "tpl_customer", "alias": "customer", "count": 20 },
    {
      "templateId": "tpl_order",
      "alias": "order",
      "count": 60,
      "relations": {
        "customerId": { "from": "customer.id", "strategy": "round-robin" }
      }
    }
  ]
}

The response echoes the seed, so a batch is as replayable as a single generate call — small batches come back synchronously with results, larger ones return a batchId to poll. Note the namespaces, which matter for replay specifically.

Does a seed make my whole test suite deterministic?

No, and it is worth being blunt about the boundary. A fixed seed pins JsonFabrica's data generation and nothing else. It does not touch your app's own sources of nondeterminism:

• wall-clock reads (Date.now(), timestamps in assertions),
• map or hash iteration order,
• unawaited promises and race conditions,
• database rows returned without an ORDER BY,
• genuine network flakiness.

What a seed does is remove the test data as a variable. Replay the exact dataset and watch what happens: if the failure still reproduces, the data wasn't the cause and you can go looking at the list above; if it stops reproducing, the data was the trigger and you now have a deterministic repro to fix against. Either way you have turned a coin-flip into a controlled experiment.

There is one more sharp edge. Sequences created with createSeq and read with getSeq are durable server-side state, scoped per namespace — the counter keeps advancing across runs and does not reset when you reuse a seed. So a replay against the default namespace draws different counter values than the original run, because the counter moved on in between. If those counter or variable values matter to the bug, run the replay under a dedicated sequenceNamespace (and variableNamespace, for getVar/setVar state) as shown in the batch above, so it starts clean and does not collide with the shared tenant-default namespace. The same advice applies to ad-hoc POST /v1/templates/generate debugging calls: set a throwaway namespace like "debug" so your probing doesn't disturb real tenant sequences. For the mechanics of counters and namespaces, see Unique Test Data Generation.

There is no magic record/replay button

To be clear about what JsonFabrica does and doesn't do: there is no built-in record/replay feature. The service does not store your past runs' seeds and cannot hand you yesterday's dataset back on its own. It returns JSON over HTTP — it doesn't run your tests, doesn't insert into your database, and isn't a test-runner plugin. The reproducibility is a property of the generator (pure function of template, seed, params) that you turn into a workflow: log meta.seed, pass it back with SEED=.... That is the whole trick, and it is why capturing the seed on every call is the one habit worth building in from day one — a point the JsonFabrica launch post makes about deterministic, reproducible generation as a first principle.

FAQ

How do I reproduce a test failure caused by random data? Log the seed from every generation response and read it back from the failed run's CI log. JsonFabrica echoes the seed it used in meta.seed even when you did not supply one, so the failing dataset is recoverable. Re-run the test locally passing that exact seed and you get byte-identical data, every field and every record, so the failure reproduces on the first try instead of one run in thirty.

Does a fixed seed make my whole test suite deterministic? No. A fixed seed pins only JsonFabrica's data generation. It does not remove your app's own nondeterminism, such as wall-clock reads, map or hash iteration order, unawaited promises, unordered database rows, or network flakiness. What it does is take the test data off the table as a variable, so if the failure still reproduces the cause is in your code or environment, and if it stops the data was the trigger.

Should I use a fixed seed or a random seed in CI? Let most CI runs pick a fresh seed so you keep surfacing new bad field combinations, but log meta.seed from every response so any failure is recoverable. Pin an explicit seed only where you want stability, such as PR gates, visual-regression, and snapshot tests. A nightly fuzz job should omit the seed entirely to maximize coverage. Capture by default, pin deliberately.

Where does JsonFabrica store the seed for a past run? It does not. There is no built-in record or replay feature, and JsonFabrica keeps no history of your runs' seeds. The seed lives in your logs because you printed meta.seed there. The workflow is entirely yours: log the seed on every generation, then pass it back with SEED=... when you want to reproduce a run.

Why do my replayed sequence counters differ from the original run? Sequences created with createSeq and read with getSeq are durable server-side state scoped per namespace, so the counter keeps advancing across runs. A replay against the default namespace draws different counter values than the original because the counter moved on in between. If counter or variable values matter to the bug, run the replay under a dedicated sequenceNamespace and variableNamespace so it starts clean and does not collide with the shared tenant default.

Is deterministic test data the same as reproducible test data? In practice yes: deterministic generation is what makes test data reproducible. Because JsonFabrica generation is a pure function of template, seed, and params, the same three inputs always produce the same output. Determinism is the property; reproducibility is the payoff, letting you regenerate an exact dataset on demand from just its seed.

JsonFabrica makes every dataset a pure function of template, seed, and params, and echoes the seed it used in meta.seed on every generate and batch response — which is all you need to capture a flaky failure and replay it byte-for-byte. Log the seed, pass it back, and turn a once-in-thirty red build into a repro you can actually fix.