Outbox under crash injection

Every design meets the same crashes. At a 20% chance, 101 of the 500 orders died before their commit, which harms nobody: nothing saved, nothing announced, and a client that heard nothing and can try again.

The dual writes break between their two writes. Committing first, 86 orders were saved and never announced, and nothing downstream will ever act on them. Publishing first, 86 orders were announced and never saved, and the consumer acted on every one: shipped, and never ordered. There is no safe order for two writes to two systems.

Writing the event to an outbox table in the order’s own transaction makes the two writes one: in the recorded run nothing was lost and nothing invented. The price is at least once. The relay died 114 times between publishing an event and marking it, and each time the event went out again; the consumer died 128 times between acting and moving on, and each time the message came again.

Deduplicating by message id, the consumer still acted on each order once. Without it, it acted 242 times too many.

Event-driven pipelines on Kafka are part of my work at Picsart, and Kafka carried event-driven processes at Worldline. Wherever a service writes to its database and announces it on a topic, this is the choice underneath: a dual write that can lose an event or invent one, or an outbox that loses nothing and may send some twice. The idempotency experiment shows the other side: a client that retries, and a server that recognises the retry.