pytest Test Data: Generated Fixtures Instead of Hand-Written Dicts

pytest test data tends to be the least designed part of a Python test suite. pytest gives you excellent plumbing: fixtures that set up and share state, conftest.py to make them available everywhere, and @pytest.mark.parametrize to run one test against a whole table of inputs. None of it says where the data should come from. So most suites end up with a few hand-written dicts in conftest.py, or with Faker calls scattered across fixtures that produce different values on every machine. This post builds a third option: a fixture file generated once from a seeded template, committed to the repo, and loaded by a session-scoped fixture. Valid cases and deliberately invalid ones share one parametrize table, every row carries its expected outcome, and any run can be reproduced from its seed.

JsonFabrica's role here is small. It's a JSON-over-HTTP API: a short Python script sends it a template and a seed, and writes the generated JSON to tests/fixtures/. There is no Python SDK and no pytest plugin, and your test runs never call the API.

Where pytest test data usually comes from

Two patterns show up in almost every Python codebase.

Hand-written dicts. A fixture returns {"id": 1, "name": "Test User", "email": "[email protected]"}, and a second one is copied from it with one field changed. The data is readable and stable, but it stays tiny. Nobody hand-writes forty customers, so tests only ever see two, and edge cases get added one dict at a time when a bug forces it.

Faker in conftest.py. A fixture calls fake.name() and fake.email(), so the data looks realistic and there's as much of it as you want. It usually brings three problems:

• It isn't reproducible unless someone seeds it. An unseeded Faker() gives different values on every run, so a failure on CI can't be replayed locally. Seeding helps, but Faker's documentation warns that output for a given seed can change between Faker versions, so a seed only reproduces data on the same pinned version.
• Related records drift apart. Each call is independent. An order's customer_id points at a customer only if the fixture code remembers to wire it up, and a "suspended customer" test only works if some fixture happens to create one.
• The data isn't reviewable. The values exist only while the tests run. You can't open a file and see which cases the suite covers.

Both patterns are fine at small scale. The trouble starts when several test modules share the same data, when the data has relationships, or when a failure on one machine has to be replayed on another.

Generate once, commit the file, read it in tests

The alternative splits the work into two steps that never run at the same time:

1. Generate offline. A regen script sends a template and a fixed seed to the JsonFabrica API and writes the response to tests/fixtures/orders.json. You run it by hand when the template changes, review the diff, and commit the file.
2. Load in tests. conftest.py reads the committed file with the standard json module. CI runs need no API key, make no network calls, and read exactly the bytes you reviewed.

The project layout for the rest of this post:

pyproject.toml           # pytest config (pythonpath)
shop/
  orders.py              # code under test
scripts/
  regen_fixtures.py      # calls the API; run by hand, not in CI
tests/
  conftest.py            # session fixture, per-test copies, report header
  fixtures/
    orders.tmpl          # the template (committed)
    orders.json          # the generated data (committed)
  test_place_order.py    # the parametrized table
  test_fixture_data.py   # checks the file's own invariants

shop needs to be importable from the tests: install the project with pip install -e ., or add pythonpath = ["."] under [tool.pytest.ini_options] in pyproject.toml (pytest 7+). Without one of those, plain pytest fails with ModuleNotFoundError: No module named 'shop'.

The code under test is a deliberately small order service with three rules: the customer must exist, the customer must be active, and the quantity must be between 1 and 100.

# shop/orders.py: the code under test, kept minimal for the example.
from dataclasses import dataclass


class OrderError(Exception):
    def __init__(self, code):
        super().__init__(code)
        self.code = code


@dataclass
class Order:
    customer_id: str
    sku: str
    quantity: int


class CustomerRepo:
    def __init__(self, customers):
        self._by_id = {c["id"]: c for c in customers}

    def get(self, customer_id):
        return self._by_id.get(customer_id)


def place_order(repo, customer_id, sku, quantity):
    customer = repo.get(customer_id)
    if customer is None:
        raise OrderError("customer_not_found")
    if customer["status"] != "active":
        raise OrderError("customer_suspended")
    if not 1 <= quantity <= 100:
        raise OrderError("quantity_out_of_range")
    return Order(customer_id, sku, quantity)

A template for pytest fixtures with realistic data

One template produces both halves of the fixture file: a list of customers, and a table of order cases that reference those customers. Keeping them in one template is what keeps them consistent. The cases can only point at customer IDs the same template just wrote.

<setVar('customers', getParam('customers', 8))>{
  "customers": [
<for(i, 1, getVar('customers'))><if(getVar('i') > 1)>,<endIf>
    {
      "id": "cus_<appendBefore('0', getVar('i'), 3)>",
      "name": "<getRandomFullName()>",
      "email": "<getRandomEmail('example.com')>",
      "status": <if(getVar('i') == 1)>"suspended"<else>"active"<endIf>,
      "createdAt": "<getRandomDate('2024-01-01', '2026-06-30')>"
    }<end_for>
  ],
  "orderCases": [
<for(i, 1, getParam('cases', 12))><if(getVar('i') > 1)>,<endIf>
    {
      "id": "case-<appendBefore('0', getVar('i'), 2)>",
      "case": <if(getVar('i') == 1)>"unknown-customer"
        <elseIf(getVar('i') == 2)>"zero-quantity"
        <elseIf(getVar('i') == 3)>"quantity-over-limit"
        <elseIf(getVar('i') == 4)>"suspended-customer"
        <else>"valid"<endIf>,
      "customerId": <if(getVar('i') == 1)>"cus_999"
        <elseIf(getVar('i') == 4)>"cus_001"
        <else>
        "cus_<appendBefore('0', getRandomNumber(2, getVar('customers')), 3)>"
        <endIf>,
      "sku": "<getRandomElement('basic', 'pro', 'pro', 'team')>",
      "quantity": <if(getVar('i') == 2)>0
        <elseIf(getVar('i') == 3)>101
        <else><getRandomNumber(1, 100)><endIf>,
      "expectedError": <if(getVar('i') == 1)>"customer_not_found"
        <elseIf(getVar('i') <= 3)>"quantity_out_of_range"
        <elseIf(getVar('i') == 4)>"customer_suspended"
        <else>null<endIf>
    }<end_for>
  ]
}

What each part does:

• customers is the only variable, because it's read twice. getParam reads the customer count from the request, defaulting to 8. The customer loop uses it as its end bound, and the valid order cases use it as the upper limit for the customer they reference. Everything else is written inline in its own field.
• IDs come from the loop index, not from randomness. appendBefore pads the index, so the customers are always cus_001 to cus_008 and the cases case-01 to case-12. Stable IDs make stable test names.
• Valid cases can only reference real, active customers. getRandomNumber(2, getVar('customers')) picks a customer from 2 up to the count, so a valid row never points at a customer that doesn't exist, and never at cus_001. Keep customers at 2 or more.
• The planted cases occupy fixed rows. Rows 1 to 4 each break exactly one rule: an unknown customer, a zero quantity, a quantity of 101, and an order from cus_001, the one customer the first loop always marks as suspended. That last case only works because the customer list and the case table come from the same template. One defect per row means an error can only have one cause.
• Every row carries its expected outcome. expectedError is the error code the row should produce, or null for a valid order.

Template expressions support comparisons (==, !=, <, <=, >, >=) and &&/||, but no arithmetic, so "exactly four invalid rows" is expressed as index bands rather than a counter. Loop bounds are inclusive, the loop variable is read with getVar('i'), and the comma guard <if(getVar('i') > 1)>,<endIf> puts a comma before every array item except the first. The control flow docs cover for and if/elseIf/else. Putting the invalid rows first means case-04 is always the suspended-customer case, however many valid rows you ask for. For a broader catalog of what to plant, such as empty strings, nulls, and limits, see boundary-value and edge-case test data.

Rendered with seed 20261008, the file the regen script writes starts like this (trimmed to two customers and five cases):

{
  "seed": 20261008,
  "customers": [
    {
      "id": "cus_001",
      "name": "Fenin Thorncroft",
      "email": "[email protected]",
      "status": "suspended",
      "createdAt": "2024-04-07T08:35:40.534Z"
    },
    {
      "id": "cus_002",
      "name": "Matthew Chopin",
      "email": "[email protected]",
      "status": "active",
      "createdAt": "2024-11-02T14:57:47.015Z"
    }
  ],
  "orderCases": [
    {
      "id": "case-01",
      "case": "unknown-customer",
      "customerId": "cus_999",
      "sku": "pro",
      "quantity": 10,
      "expectedError": "customer_not_found"
    },
    {
      "id": "case-02",
      "case": "zero-quantity",
      "customerId": "cus_008",
      "sku": "pro",
      "quantity": 0,
      "expectedError": "quantity_out_of_range"
    },
    {
      "id": "case-03",
      "case": "quantity-over-limit",
      "customerId": "cus_002",
      "sku": "pro",
      "quantity": 101,
      "expectedError": "quantity_out_of_range"
    },
    {
      "id": "case-04",
      "case": "suspended-customer",
      "customerId": "cus_001",
      "sku": "pro",
      "quantity": 85,
      "expectedError": "customer_suspended"
    },
    {
      "id": "case-05",
      "case": "valid",
      "customerId": "cus_005",
      "sku": "pro",
      "quantity": 86,
      "expectedError": null
    }
  ]
}

Notice that the invalid rows are otherwise ordinary. case-02 uses a real, active customer, so the only thing wrong with it is the quantity. case-04 has a valid quantity, so the only thing wrong with it is the customer's status.

Generate test data in Python with a regen script

The regen script is the only code that talks to JsonFabrica. It uses requests, sends the raw template to POST /v1/templates/generate, and writes the result. That endpoint renders a template without saving it, so the template stays a file in your repo, next to the data it produces.

#!/usr/bin/env python3
"""Regenerate tests/fixtures/orders.json from orders.tmpl.

Run by hand when the template changes, then review and commit the
result. CI never runs this script; it only reads the committed file.
"""
import argparse
import json
import os
from pathlib import Path

import requests

API_URL = "https://api.jsonfabrica.com/v1/templates/generate"
FIXTURES = Path(__file__).resolve().parents[1] / "tests" / "fixtures"
DEFAULT_SEED = 20261008  # change only when you mean to replace the data


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--seed", type=int, default=DEFAULT_SEED)
    parser.add_argument(
        "--random-seed",
        action="store_true",
        help="omit the seed and let the API pick one",
    )
    parser.add_argument("--cases", type=int, default=12)
    args = parser.parse_args()

    request = {
        "body": (FIXTURES / "orders.tmpl").read_text(),
        "params": {"customers": 8, "cases": args.cases},
    }
    if not args.random_seed:
        request["seed"] = args.seed

    api_key = os.environ["JSONFABRICA_API_KEY"]
    response = requests.post(
        API_URL,
        json=request,
        headers={"Authorization": f"Bearer {api_key}"},
        timeout=30,
    )
    if not response.ok:
        # The body is {"error": {"code", "message", "details"}}, e.g.
        # INVALID_TEMPLATE, UNKNOWN_FUNCTION, or GENERATION_FAILED.
        raise SystemExit(f"{response.status_code}: {response.text}")
    result = response.json()

    seed = result["meta"]["seed"]
    if not args.random_seed and seed != args.seed:
        raise SystemExit(f"API used seed {seed}, expected {args.seed}")

    out = FIXTURES / "orders.json"
    payload = {"seed": seed, **result["data"]}
    out.write_text(json.dumps(payload, indent=2) + "\n")
    cases = len(payload["orderCases"])
    print(f"wrote {out} (seed {seed}, {cases} cases)")


if __name__ == "__main__":
    main()

The details that matter:

• The response is { data, meta }. data is the generated document, already parsed as JSON. meta.seed echoes the seed the API used, which is the one you sent or, if you sent none, one it picked.
• The seed goes into the file. Writing meta.seed as a top-level seed key means the fixture file records how to rebuild itself. meta.generatedAt is left out on purpose: it changes on every call and would make every regeneration show a diff.
• The seed check stops silent drift. If the seed were dropped by mistake, the API would pick a random one, and you'd commit a file nobody can regenerate.
• Same template, seed, and params give the same file. Running the script twice produces no diff. Raising --cases appends rows and leaves the existing ones as they were, because the case count doesn't consume randomness. Changing the customer count or editing the template shifts the random draws, so commit the regenerated file in the same pull request as the template change.

Reproducible test data from seeds covers how seeding works in more depth.

Organizing conftest.py: a session-scoped fixture for shared data

conftest.py is where shared fixtures belong. pytest discovers it automatically, and fixtures defined in tests/conftest.py are available to every test module in tests/ and its subdirectories, without imports. Don't import from conftest.py in test modules; let pytest inject fixtures by name. If one area of the suite needs its own fixtures, give that subdirectory its own conftest.py.

# tests/conftest.py
import copy
import json
from pathlib import Path

import pytest

from shop.orders import CustomerRepo

FIXTURE_FILE = Path(__file__).parent / "fixtures" / "orders.json"


def pytest_report_header(config):
    seed = json.loads(FIXTURE_FILE.read_text())["seed"]
    return f"orders fixture: {FIXTURE_FILE.name}, seed {seed}"


@pytest.fixture(scope="session")
def fixture_data():
    """The generated fixture file, read once per test session."""
    return json.loads(FIXTURE_FILE.read_text())


@pytest.fixture
def customers(fixture_data):
    """A fresh copy per test, so no test can change shared data."""
    return copy.deepcopy(fixture_data["customers"])


@pytest.fixture
def customer_repo(customers):
    return CustomerRepo(customers)

How the pieces fit:

• fixture_data is session-scoped. It reads and parses the file once per test run, and every test that requests it gets the same object. Under pytest-xdist each worker process is its own session, so each worker reads the file once. That's cheap for a local file, and one reason not to call an API from a session fixture.
• Tests get copies, not the shared object. A session-scoped dict is shared by every test in the run. If one test edits a customer's status, every later test sees the edit, and results start depending on test order. The function-scoped customers fixture deep-copies the list for each test. A function-scoped fixture can depend on a session-scoped one; the reverse fails with a scope mismatch error.
• The path is built from __file__. Path(__file__).parent is the directory containing conftest.py, so the fixture works whether pytest is started from the repo root, from tests/, or by an IDE.
• pytest_report_header prints the seed. This hook adds a line to the header pytest prints at the start of every session, so each CI log records which data the run used. That pays off when you need to reproduce a run, covered below.

Because the customers and cases come from one template, the file has invariants a test can check. A short test catches a template edit that breaks them before it shows up as a confusing failure elsewhere:

# tests/test_fixture_data.py
def test_valid_cases_reference_active_customers(fixture_data):
    status = {c["id"]: c["status"] for c in fixture_data["customers"]}
    for case in fixture_data["orderCases"]:
        if case["expectedError"] is None:
            assert status.get(case["customerId"]) == "active", case["id"]

If the same fixture file also has to seed a real database, the Testcontainers test data post shows a pytest fixture that loads generated JSON into a throwaway Postgres container.

pytest parametrize test data: valid rows and planted invalid rows

The order cases become a parametrize table. Every row is one test, and because each row says what should happen, one test function covers both the valid orders and the planted invalid ones:

# tests/test_place_order.py
import json
from pathlib import Path

import pytest

from shop.orders import OrderError, place_order

FIXTURE_FILE = Path(__file__).parent / "fixtures" / "orders.json"
CASES = json.loads(FIXTURE_FILE.read_text())["orderCases"]


@pytest.mark.parametrize(
    "case", CASES, ids=[f"{c['id']}-{c['case']}" for c in CASES]
)
def test_place_order(case, customer_repo):
    args = (case["customerId"], case["sku"], case["quantity"])

    if case["expectedError"] is None:
        order = place_order(customer_repo, *args)
        assert order.customer_id == case["customerId"]
        assert order.quantity == case["quantity"]
    else:
        with pytest.raises(OrderError) as excinfo:
            place_order(customer_repo, *args)
        assert excinfo.value.code == case["expectedError"]

A few things are worth spelling out:

• The table is read at module level, not from a fixture. pytest evaluates the arguments to @pytest.mark.parametrize when it collects the module, before any fixture has run, so the decorator can't take fixture_data. Reading the same file with a plain json.loads at import time is the simple fix. The test still uses the customer_repo fixture for the shared customers.

• ids= makes failures readable. Without it, pytest names the tests case0, case1, and so on. With it, pytest -v prints names that say what each row is:

  tests/test_place_order.py::test_place_order[case-01-unknown-customer] PASSED
  tests/test_place_order.py::test_place_order[case-04-suspended-customer] PASSED
  tests/test_place_order.py::test_place_order[case-05-valid] PASSED
  

  Because the IDs come from the template's loop index, they are stable across runs, so pytest -k suspended or a node ID such as "tests/test_place_order.py::test_place_order[case-04-suspended-customer]" always selects the same case.

• Invalid rows must fail for the right reason. Checking excinfo.value.code against expectedError catches the bug where the quantity check accidentally fires on the suspended-customer row. A test that only checked "some error was raised" would pass there.

The same idea, a data table where each row carries the result it expects, also drives API runs outside pytest; the Postman test data post applies it to a Newman data file.

Sharing the table across modules with pytest_generate_tests

If several test modules need the same cases, reading the file in each one gets repetitive. A pytest_generate_tests hook in conftest.py can parametrize any test, in any module under tests/, that asks for an order_case argument:

# tests/conftest.py (continued)
def pytest_generate_tests(metafunc):
    if "order_case" in metafunc.fixturenames:
        cases = json.loads(FIXTURE_FILE.read_text())["orderCases"]
        ids = [f"{c['id']}-{c['case']}" for c in cases]
        metafunc.parametrize("order_case", cases, ids=ids)

A test then declares def test_something(order_case, customer_repo): and gets one run per row, with no decorator and no file handling.

If each row needs setup before the test sees it, such as turning the dict into a request object, indirect parametrization is the alternative. Use it instead of the pytest_generate_tests hook, not alongside it: the hook would parametrize order_case a second time and pytest would refuse to collect the test. With the hook removed, @pytest.mark.parametrize("order_case", CASES, indirect=True), where CASES is the module-level list loaded in test_place_order.py, passes each row to a fixture named order_case, which reads it as request.param and returns whatever the test should receive.

Reproduce a failing run from its seed

With a committed file, most reproduction is trivial. A failure in CI used the same orders.json you have locally, so the node ID from the CI log reruns the exact case:

pytest "tests/test_place_order.py::test_place_order[case-07-valid]"

The seed earns its keep when you go looking for data you didn't plan for. A nightly exploratory job can regenerate the fixture with a seed the API picks, run the suite against that data, and leave the committed file alone. The job uses the API and an API key, so keep it separate from the CI runs that gate merges:

python scripts/regen_fixtures.py --random-seed --cases 200
pytest

The regen script prints the seed it received, and the report header prints it again at the top of the pytest output:

orders fixture: orders.json, seed 2214596771

When the nightly job fails, take the seed from its log and regenerate locally with the same parameters, from the same commit so the template matches:

python scripts/regen_fixtures.py --seed 2214596771 --cases 200
pytest

Same template, same seed, same parameters: the file is identical to the one the nightly job tested, and the failure reproduces. The seed in that example is a placeholder; use the one from your own log.

Once you've found the bug, decide what to keep. Usually the failing row points at a case the planted rows didn't cover. Add it as a new planted row in the template, regenerate with the default seed, and commit the template and data together. If you're not changing the template, git restore tests/fixtures/orders.json puts the committed file back.

When Faker or factory_boy is the better choice

Generate-once-and-commit is extra machinery: a template, a script, an API key for whoever regenerates, and a JSON file to review. For a lot of tests, that's more than the job needs.

Use Faker when the data is small, throwaway, and unasserted. A signup test that needs one unique email, a fixture that fills a required name field nobody checks, a quick script that populates a dev database: fake.email() inline is simpler than any file. If you want repeatability, Faker.seed(1234) seeds it, and Faker's pytest plugin provides a faker fixture that is seeded with 0 unless you define a faker_seed fixture. Pin the exact Faker version if any assertion depends on specific generated values, since Faker's own documentation says output for a seed can change between releases.

Use factory_boy when you're building ORM objects in Python. factory_boy declares factories with factory.Faker fields, factory.Sequence for unique values, and factory.SubFactory for related objects, so an OrderFactory() creates its own customer. DjangoModelFactory and SQLAlchemyModelFactory save straight through your models. If every test builds the objects it needs, and the relationships are parent-child ones that SubFactory expresses well, factory_boy is a better fit than a JSON file. factory.random.reseed_random() seeds both factory_boy and Faker when you need repeatable values.

Generate and commit when the data is shared or has to be replayed. The file-based approach wins when:

• Several test modules, or several projects, share one data set. The JSON file can also feed a frontend's tests or a Postman collection, which a Python factory can't.
• Runs must reproduce across machines and CI. The committed bytes are the data. There is no dependency on a Faker version, on test order, or on how many values another test drew first.
• Records must stay consistent across a whole set. A suspended customer that exactly one case targets, valid cases that only point at active customers: these are properties of the whole data set, set in one template, not of one object built in isolation.
• Edge cases are planted and reviewed. The invalid rows sit in a file a reviewer can read in the pull request, each with its expected error, instead of being spread across factory traits and test bodies.

If what you actually want is a tool that searches for inputs that break a function, that's property-based testing, and Hypothesis is the standard Python library for it. It complements a fixed case table rather than replacing it.

The approaches mix well. A suite can load its shared customers and its case table from the committed file, and still use Faker inside a single test for a throwaway value nobody asserts on.

FAQ

How do I share fixtures across multiple test files in pytest?

Define them in a conftest.py file. pytest discovers conftest.py automatically, and fixtures defined in it are available to every test module in the same directory and its subdirectories, without any import. Put fixtures used by the whole suite in tests/conftest.py, and fixtures used by one area in a conftest.py inside that subdirectory.

How do I load test data from a JSON file in pytest?

Read the file in a fixture with json.loads(path.read_text()) and return the result. Build the path from the conftest file's own location, for example Path(__file__).parent / "fixtures" / "orders.json", so it works no matter which directory pytest is started from. Use scope="session" to read it once per test run, and hand each test a copy if tests might modify the data.

Can you use a fixture inside pytest.mark.parametrize?

Not directly. The values passed to @pytest.mark.parametrize are evaluated when pytest collects and imports the test module, before any fixture has run. Either load the data at module level with a plain function, generate the parameters in a pytest_generate_tests hook in conftest.py, or pass indirect=True so each value is handed to a fixture as request.param.

How do I make Faker data reproducible in pytest?

Call Faker.seed() with a fixed value before generating, or use the faker fixture from Faker's pytest plugin, which is seeded with 0 unless you define a faker_seed fixture. The same seed then gives the same values on the same Faker version. Faker's documentation warns that results can change between versions as its datasets are updated, so pin the exact version if your assertions depend on specific values.

Does JsonFabrica have a Python SDK or pytest plugin?

No. JsonFabrica is an HTTP API that returns generated JSON. You call it from a small script with requests, httpx, or curl, save the generated data as a JSON file in your repository, and load that file in your pytest fixtures with the standard json module. Your test runs never need to call the API.

A seeded template, a short regen script, and one committed file give your pytest suite shared fixtures with realistic data, a parametrize table with planted invalid cases, and runs you can replay from a seed. Seeded, parameterized generation is part of the JsonFabrica API, and the templates API reference documents the generate request and response.