Kafka Test Data: Event Sequences That Catch Consumer Bugs

A Kafka consumer is rarely wrong about a single message. It goes wrong when OrderShipped arrives before OrderPlaced, when the same OrderPaid is delivered twice, or when a producer that deployed an hour early starts sending version 2 payloads into a stream your consumer still reads as version 1. Useful Kafka test data is therefore a sequence, not a sample message: events that belong to the same orders, in a known order, with each awkward case planted at a fixed position.

This post builds that sequence from one seeded template: eight orders, each going through OrderPlaced, OrderPaid and OrderShipped, seven of them with one planted problem. Then it produces the stream to a test topic and asserts your consumer's end state. JsonFabrica generates the JSON. Converting it to lines and producing it to Kafka is your own tooling, shown here with kcat and a Testcontainers test.

Why single-message mocks miss consumer bugs

A webhook handler test can get a long way with one payload at a time, as in mock webhook payloads. A stream consumer keeps state between messages: a projection of each order, the event ids it has processed, and which transitions are allowed. Most bugs in that state only show up across several messages for the same entity:

• Out-of-order events: a late OrderPlaced resets a shipped order to PLACED.
• Duplicate delivery: an at-least-once redelivery records the payment twice.
• Schema-version drift: a v1-only consumer reads a renamed v2 field as a zero total.
• Unknown event types: a new event type crashes the consumer instead of being skipped.
• Missing optional fields: an absent field throws a null pointer exception.
• Poison messages: one malformed message, retried forever, blocks its partition.

None of this shows up in a test that sends one hand-written message.

Plan the Kafka test data: one planted case per order

Decide the cases before writing the template. Each order gets at most one problem, so a failing assertion points at one cause:

That's 24 regular events plus 3 extra messages: 27 in total. Two v2 orders instead of one make the drift a mix, not a cutover, which is what a rolling deploy produces. Putting the poison message before ORD-0008 means order 8 can only reach SHIPPED if the consumer got past it.

This is the same plan-then-assert pattern as ETL test data, which plants defects in batch tables. Here the defects live in the order and repetition of messages rather than in rows.

Generate event payloads that stay consistent per order

The template loops over orders, and for each order loops over three stages. It returns a JSON array where each item is one Kafka message: a key and a value. The poison message is the exception, with a raw string instead of a value, for reasons covered below.

[<for(i, 1, getParam('orders', 8))>
<setVar('orderNo', appendBefore('0', getVar('i'), 4))>
<setVar('total', getRandomNumber(15, 400, 2))>
<setVar('paidAt', getRandomDate('2026-09-01T10:00Z', '2026-09-01T12:00Z'))>
<for(s, 1, 3)>
<if(getVar('i') == 3 && getVar('s') == 1)><setVar('type', 'OrderShipped')>
<elseIf(getVar('i') == 3 && getVar('s') == 3)><setVar('type', 'OrderPlaced')>
<elseIf(getVar('s') == 1)><setVar('type', 'OrderPlaced')>
<elseIf(getVar('s') == 2)><setVar('type', 'OrderPaid')>
<else><setVar('type', 'OrderShipped')><endIf>
<if(getVar('i') > 1 || getVar('s') > 1)>,<endIf>
{"key": "ORD-<getVar('orderNo')>", "value": {
  "eventId": "evt-<getVar('orderNo')>-<toLowerCase(getVar('type'))>",
  "type": "<getVar('type')>",
  "orderId": "ORD-<getVar('orderNo')>",
<if(getVar('type') == 'OrderPlaced' && (getVar('i') == 4 || getVar('i') == 8))>
  "schemaVersion": 2,
  "occurredAt": "<getRandomDate('2026-09-01T08:00Z', '2026-09-01T10:00Z')>",
  "customerEmail": "<getRandomEmail('example.com')>",
  "amount": {"value": <getVar('total')>, "currency": "EUR"},
  "channel": "<getRandomElement('web', 'ios', 'android')>"
<elseIf(getVar('type') == 'OrderPlaced')>
  "schemaVersion": 1,
  "occurredAt": "<getRandomDate('2026-09-01T08:00Z', '2026-09-01T10:00Z')>",
  "customerEmail": "<getRandomEmail('example.com')>",
  "total": <getVar('total')>,
  "currency": "EUR"
<elseIf(getVar('type') == 'OrderPaid')>
  "schemaVersion": 1,
  "occurredAt": "<getVar('paidAt')>",
  "amount": <getVar('total')>,
  "paymentId": "pay_<getVar('orderNo')>"
<else>
  "schemaVersion": 1,
  "occurredAt": "<getRandomDate('2026-09-01T12:00Z', '2026-09-01T18:00Z')>",
  "carrier": "<getRandomElement('DHL', 'UPS', 'GLS')>"<if(getVar('i') != 2)>,
  "trackingNumber": "<getRandomNumber(100000000, 999999999)>"<endIf>
<endIf>
}}<end_for>
<if(getVar('i') == 5)>,
{"key": "ORD-<getVar('orderNo')>", "value": {
  "eventId": "evt-<getVar('orderNo')>-orderpaid",
  "type": "OrderPaid",
  "orderId": "ORD-<getVar('orderNo')>",
  "schemaVersion": 1,
  "occurredAt": "<getVar('paidAt')>",
  "amount": <getVar('total')>,
  "paymentId": "pay_<getVar('orderNo')>"
}}<endIf>
<if(getVar('i') == 6)>,
{"key": "ORD-<getVar('orderNo')>", "value": {
  "eventId": "evt-<getVar('orderNo')>-ordergiftwrapped",
  "type": "OrderGiftWrapped",
  "orderId": "ORD-<getVar('orderNo')>",
  "schemaVersion": 1,
  "occurredAt": "<getRandomDate('2026-09-01T10:00Z', '2026-09-01T12:00Z')>"
}}<endIf>
<if(getVar('i') == 7)>,
{"key": "ORD-<getVar('orderNo')>",
 "raw": "{\"eventId\":\"evt-<getVar('orderNo')>-orderpaid\",\"type\":\"Ord"}
<endIf>
<end_for>]

Only four values are stored with setVar, and each one is read in more than one place:

• orderNo is the zero-padded loop index, so order 3 is 0003. It goes into the key, the orderId, the eventId and the paymentId of every event for that order. This is what keeps the stream consistent per entity: there's no way for OrderPaid to name a different order than the OrderPlaced before it.
• total is read by OrderPlaced and OrderPaid, so the payment always matches the order. It's set at the top of the order rather than inside OrderPlaced because for order 3, OrderPaid is generated before OrderPlaced.
• paidAt is read by OrderPaid and again by order 5's duplicate, which has to repeat the original byte for byte.
• type is chosen per stage and decides the event id, the type field and which payload fields follow.

Everything else, such as the email, the carrier and the tracking number, is a plain inline call. The outer loop's bound is getParam with a default of 8, so you can ask for more orders without editing the template.

Each planted case is an index condition from the control flow syntax. The loop variables are read with getVar('i') and getVar('s'); a bare i inside a condition is just the string "i". Template expressions support ==, !=, <, <=, >, >=, &&, ||, ! and parentheses, which is all the planting needs:

• Out of order (order 3). The first two lines of the type chain swap stages 1 and 3. Each event still gets the timestamp of its own type, which matters in the next section.
• Duplicate (order 5). A second OrderPaid after the stage loop, with the same event id, timestamp, amount and payment id. Coming after OrderShipped, it would be an illegal transition if treated as new.
• Version drift (orders 4 and 8). A separate OrderPlaced branch.
• Missing optional field (order 2). The trackingNumber line, with its leading comma, sits inside <if(getVar('i') != 2)>, so the JSON stays valid when it's left out.
• Unknown type and poison (orders 6 and 7). Extra messages after the stage loop.

The comma guard writes a comma before every event except the very first; the extra messages always follow an event, so their comma is unconditional.

The poison message is a string because the engine parses what it renders and fails the request if it isn't valid JSON. So the malformed body is carried as the JSON string "raw", and the conversion step below writes it out verbatim. A poison message that is valid JSON but missing orderId would be an ordinary value without that field.

Timestamps when the template can't do arithmetic

There's no arithmetic in templates, so you can't write "paid five minutes after placed". Instead, each stage draws getRandomDate from its own range, and the ranges don't overlap:

So for every order, placed comes before paid and paid before shipped, whatever the seed. A late event is simulated by where its line sits in the stream, not by an earlier timestamp. Order 3's OrderPlaced carries a timestamp from the 08:00-10:00 placed window, but it's the third message for that order, so the consumer sees an occurredAt older than events it has already processed.

If your schema uses epoch milliseconds, generate them directly with getRandomNumber and integer bounds: getRandomNumber(1788249600000, 1788256800000) returns a whole number in the same 08:00-10:00Z window. Store it with setVar if more than one event reads it, as paidAt is. What the template can't do is convert getRandomDate's ISO output to epoch, so if you keep ISO strings in the template, convert them in your jq step or in the producer.

Render the event stream test data with a fixed seed

Save the template once with POST /v1/templates, then render it with POST /v1/templates/{templateId}/generate, passing a seed and the number of orders:

SEED=20261012
curl -sS --fail -X POST \
  "https://api.jsonfabrica.com/v1/templates/$EVENTS_TEMPLATE_ID/generate" \
  -H "Authorization: Bearer $JSONFABRICA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"seed\": $SEED, \"params\": {\"orders\": 8}}" \
  -o response.json

# Stop if the seed wasn't applied: nobody could regenerate the file.
jq -e --argjson s "$SEED" '.meta.seed == $s' response.json > /dev/null

The response is { "data": [...], "meta": { "seed", "templateId", "generatedAt" } }, with the 27 messages in data. The same seed and the same orders value return the same messages every time, so commit the converted file next to your tests, and CI can run without an API key. Deterministic test data from seeds covers capturing and replaying seeds in general.

Rendered with seed 20261012, order 3 arrives like this:

and order 4's v2 OrderPlaced value comes out as:

{
  "eventId": "evt-0004-orderplaced",
  "type": "OrderPlaced",
  "orderId": "ORD-0004",
  "schemaVersion": 2,
  "occurredAt": "2026-09-01T09:40:50.090Z",
  "customerEmail": "[email protected]",
  "amount": { "value": 103.9, "currency": "EUR" },
  "channel": "ios"
}

JsonFabrica stops at that JSON response: no JSON Lines files, no Kafka connector. One jq command turns the array into one message per line, with the key, a tab, then the compact value or the poison message's raw string:

jq -r '.data[] | "\(.key)\t\(.raw // (.value | tojson))"' \
  response.json > src/test/resources/order-events.tsv

The result is 27 lines. The poison line is ORD-0007 followed by a tab and {"eventId":"evt-0007-orderpaid","type":"Ord, and the duplicate for order 5 is identical to the original OrderPaid line. If you don't need keys, jq -c '.data[]' response.json gives plain JSON Lines with one envelope per line.

The template stays within the engine's per-request limits up to about 320 orders, just under 1,000 events, which is plenty for a consumer test.

Keys, partitions, and what Kafka will and won't reorder

Kafka orders messages within a partition, not across a topic. The default partitioner hashes the key to choose a partition, so all messages with the same key go to the same partition, as long as the topic's partition count doesn't change. All of an order's events share the order id as key, so a consumer sees them in the order the broker received them. Without a key, events for the same order can land on different partitions and be consumed in any order. That affects each planted case differently:

• Duplicates can be planted freely. In a real system they come from redelivery after a crash or rebalance, or from an application that resends after a timeout. A resent copy has a different offset, which is why the consumer should deduplicate on eventId, not the offset.
• Poison messages can be planted freely too. The broker stores bytes and doesn't check that a message is valid JSON.
• Out-of-order events are the case to be careful with. Kafka never reorders messages from one producer within a partition when the idempotent producer is on. The Java client turns it on by default since Kafka 3.2 (and 3.0.1/3.1.1), unless you set acks below all or retries=0, which turn it off with only an INFO log. More than 5 in-flight requests also turned it off before Kafka 4.0, with a WARN log; from 4.0 the producer refuses to start instead. librdkafka-based tools such as kcat leave it off by default. Without idempotence and with max.in.flight.requests.per.connection above 1, a retried batch can land behind a later one. Otherwise, out-of-order events in production come from upstream: two services publishing to different topics, or a producer keyed by something other than the order id. In this test, order 3's lines are written in the wrong order, and because they share a key, Kafka delivers them in that wrong order every time.

Planting the reordering in the file keeps the test reproducible. With different keys or topics, the interleaving would depend on how the consumer polls its partitions, and the test could flake.

Use a test topic with one partition for the main run. All 27 messages then arrive in file order, and the last line, order 8's OrderShipped, is the last message the consumer sees. A second run against a topic with several partitions checks that the consumer doesn't rely on ordering across different orders, which Kafka doesn't promise either. That run needs a different wait condition, because order 8's OrderShipped is no longer guaranteed to be the last message consumed: wait until all 8 orders are SHIPPED and both dead letters are present.

Produce the events to a test topic

Any producer that reads a key and a value per line will do. With kcat (formerly kafkacat), -P produces, and -K sets the delimiter between key and value on each input line. -X enable.idempotence=true turns on the idempotent producer, which librdkafka leaves off by default; without it, a retry can put messages on the topic out of file order:

kcat -P -b localhost:9092 -t orders.events -K $'\t' \
  -X enable.idempotence=true \
  < src/test/resources/order-events.tsv

With kafka-console-producer.sh, set parse.key=true and key.separator to a tab (--reader-property on Kafka 4.2+, --property before that).

For an automated test, produce the file from the test itself. This JUnit 5 example uses the Testcontainers Kafka module (org.testcontainers.kafka.KafkaContainer, Testcontainers 1.20.1+ for apache/kafka-native, imports omitted). Container lifecycle, singletons and reuse are covered in Testcontainers test data. The part that matters here is reading the file and sending it in order:

@Testcontainers
class OrderConsumerStreamTest {

  @Container
  static final KafkaContainer kafka =
      new KafkaContainer("apache/kafka-native:3.8.0");

  static final String TOPIC = "orders.events";

  @BeforeAll
  static void createTopic() throws Exception {
    try (Admin admin = Admin.create(Map.of(
        AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
        kafka.getBootstrapServers()))) {
      admin.createTopics(List.of(new NewTopic(TOPIC, 1, (short) 1)))
          .all().get();
    }
  }

  static void produceFixture() throws Exception {
    Map<String, Object> config = Map.of(
        ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
        kafka.getBootstrapServers(),
        ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
        StringSerializer.class,
        ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
        StringSerializer.class);
    Path fixture = Path.of("src/test/resources/order-events.tsv");
    try (var producer = new KafkaProducer<String, String>(config)) {
      for (String line : Files.readAllLines(fixture)) {
        String[] kv = line.split("\t", 2);
        producer.send(new ProducerRecord<>(TOPIC, kv[0], kv[1])).get();
      }
    }
  }
}

Waiting on each send with .get() keeps the file's order exactly, and is fast enough for 27 messages. By default the producer stamps each record with the send time, so Kafka's record timestamps follow file order, not occurredAt. If your consumer judges lateness from record timestamps, pass explicit timestamps to ProducerRecord. Start your consumer the way the application does, pointed at kafka.getBootstrapServers(), for example through @DynamicPropertySource in a Spring Boot test.

Test Kafka consumers with synthetic events: assert the end state

Once the stream is consumed, assert the state it should have produced. Every expected value below comes from the plan table or the committed file. orders, payments and readDeadLetters stand for your own repositories and helpers, and AssertJ and Awaitility do the rest:

@Test
void consumerHandlesThePlantedStream() throws Exception {
  produceFixture();

  // One partition: order 8's OrderShipped is the last message, and
  // the poison message comes before it.
  await().atMost(Duration.ofSeconds(30)).until(() ->
      orders.find("ORD-0008").map(Order::status).orElse(null) == SHIPPED);

  // Out of order: OrderPlaced arrived last, the order still shipped.
  assertThat(orders.get("ORD-0003").status()).isEqualTo(SHIPPED);

  // Duplicate: one payment, total not doubled, no error.
  assertThat(payments.findByOrder("ORD-0005")).hasSize(1);
  assertThat(orders.get("ORD-0005").paidTotal())
      .isEqualByComparingTo("36.95");

  // Version drift: v2 totals read from amount.value.
  assertThat(orders.get("ORD-0004").total())
      .isEqualByComparingTo("103.90");
  assertThat(orders.get("ORD-0008").total())
      .isEqualByComparingTo("269.96");

  // Missing optional field: shipped, no tracking number.
  assertThat(orders.get("ORD-0002").status()).isEqualTo(SHIPPED);
  assertThat(orders.get("ORD-0002").trackingNumber()).isNull();

  // Every order reached SHIPPED.
  assertThat(orders.all()).hasSize(8)
      .allMatch(o -> o.status() == SHIPPED);

  // Dead letters: the poison and the unknown type, not the duplicate.
  assertThat(readDeadLetters("orders.events.dlq"))
      .extracting(DeadLetter::key, DeadLetter::reason)
      .containsExactlyInAnyOrder(
          tuple("ORD-0006", "unknown_type"),
          tuple("ORD-0007", "unparseable"));
}

What each group checks:

• Idempotency. Order 5's second OrderPaid has an eventId the consumer has already processed, so it should be acknowledged and skipped. A consumer that validates the transition before checking for duplicates sees PAID after SHIPPED, treats it as an error and dead-letters it. The last assertion catches that, because the duplicate must not appear in the dead-letter topic.
• Ordering. The expected state for order 3 depends on your policy. The assertions above assume the consumer parks events for orders it hasn't seen and replays them once OrderPlaced arrives. If your policy is to reject early events to a retry topic, assert that instead. What should never happen is order 3 ending in PLACED.
• Dead-letter routing. Kafka has no broker-side dead letter queue for regular consumer groups (share groups, Kafka's queue-style consumers, get one in a future release via KIP-1191), so your consumer, or a framework such as Spring Kafka, has to publish the failed message there and commit past it. Order 8 reaching SHIPPED proves the consumer didn't get stuck retrying order 7's poison message. If your policy for unknown types is to log and skip, drop ORD-0006 from the expected list.

Because the cases are planted at fixed indices, changing the seed changes the emails, amounts and timestamps but never which order has which problem. Only the exact totals in the assertions are tied to the committed file.

FAQ

How do I generate test data for Kafka?

Generate the event payloads as JSON from a seeded template, convert them to one message per line with the message key in front, and produce that file to a test topic with kcat, kafka-console-producer.sh, or a small producer in your test code. Make the events consistent per entity, so the same orderId runs through OrderPlaced, OrderPaid and OrderShipped, and plant the awkward cases, such as duplicates, out-of-order events and malformed messages, at fixed positions so you know the expected end state before the test runs.

Does Kafka guarantee message order?

Only within a single partition. Messages with the same key go to the same partition under the default partitioner, as long as the partition count doesn't change, so all events for one order arrive in the order the broker received them if they share the order id as key. Kafka makes no ordering promise across partitions, across topics, or for messages produced in the wrong order in the first place.

How do you test that a Kafka consumer is idempotent?

Produce the same message twice, with the same event id and payload, and assert that the side effect happened once: one payment row, one email, totals not doubled. Put the second copy after a later event for the same entity as well as right after the first, because a consumer that checks state transitions before checking for duplicates will reject a late copy as an illegal transition. Deduplicate on an event id in the payload, not on the Kafka offset, since a resent copy of a message gets a new offset.

Does Kafka have a dead letter queue?

Not for regular consumer groups. There, a dead-letter topic is an ordinary topic that your consumer, or a framework around it, writes failed messages to (share groups, Kafka's queue-style consumers, get a broker-side DLQ in a future release via KIP-1191). Kafka Connect sink connectors and client libraries such as Spring Kafka have dead-letter handling built in, but a plain consumer has to catch the failure, publish the message to the dead-letter topic itself, and then commit past it.

Can JsonFabrica write events to Kafka or produce Avro or Protobuf?

No. JsonFabrica returns generated JSON over HTTP. It has no Kafka or broker connector, doesn't produce messages, doesn't write JSON Lines files, and doesn't support Schema Registry, Avro or Protobuf. You convert the JSON response to lines with a tool such as jq and produce it with your own tooling.

One seeded template gives you a Kafka event stream that's consistent per order, with every sequence bug your consumer has to survive planted where you can name it. Seeded templates with loops, index conditions and parameters are part of the JsonFabrica API, and the templates API reference documents the generate request and response.