Two Writes That Must Never Come Apart

Part 8 of the "Behind the Map" series — how a review and the message that tells the rest of the system about it are guaranteed to succeed together or fail together

Abstract

Every incoming review has to trigger two things at once: get saved — and notify the rest of the system that it exists. If the process crashes exactly between these two steps, the worst case is a saved review nobody else ever learns about, or a notification about a review that was never actually saved. This article shows the challenge, three approaches we weighed against each other, and the so-called outbox pattern we settled on.

1. The challenge

When a new review arrives, two things ideally happen: it gets stored permanently, and an "this review is new" event gets recorded for other parts of the system — a later analysis run, say. Both steps belong together as one logical event, but are technically two separate actions. Exactly between those two actions, a process can crash, a server can restart, a network connection can drop.

Three outcomes are possible, and two of them are bugs:

• Both succeed. The desired case.
• The review gets saved, the notification is lost. The review exists, but no downstream process ever learns about it — a silent blind spot.
• The notification goes out, the review never gets saved. A downstream process reacts to something that doesn't actually exist in the system.

The real challenge is: how do you guarantee that only the first case ever happens — without introducing a second, complicated system (a message server, an external queue) just to get that one guarantee?

2. Approaches considered

The key insight: the database itself already provides the reliability guarantee — it just has to be used for two things instead of one.

3. The conceptual solution

Instead of sending the notification directly and immediately to an external system, it first gets stored merely as another record in the same database transaction that also saves the review — with a status of "still pending." Saving the review and recording the notification are then technically a single, indivisible action: either both succeed, or neither does. A second, completely independent process later works regularly through the pending notifications and actually delivers them — and it's exactly that second step that's allowed to fail and be retried later, without anything ever getting lost, because the pending notification is already safely stored.

4. The technical solution in detail

sequenceDiagram
    participant A as Incoming review
    participant DB as One database transaction
    participant O as Outbox table
    participant D as Delivery run (separate, later)
    A->>DB: Save (review + event "pending")
    Note over DB: Both succeed together — or both fail together
    DB->>O: Event stored safely, status "pending"
    Note over O,D: Any amount of time can pass in between
    D->>O: Claim up to 500 pending events
    D->>D: Attempt delivery
    alt Delivery succeeds
        D->>O: Status "sent"
    else Delivery fails
        D->>O: Status "failed" — the batch keeps going regardless
    end
    Note over O: Delivered events older than 30 days get purged

Step 1 — the same transaction. Saving a review and recording its event run over the same database connection, within the same unit of work from part 7 of this series. That means the "both or neither" guarantee of a single transaction automatically covers both steps together — with no new technology required to get it.

Step 2 — the outbox table. Every event first sits with a "pending" status in its own table. It hasn't been delivered yet — but it's already safe, because it was part of the same transaction that also touched the review itself.

Step 3 — a separate, fault-tolerant delivery run. In a second, independent step, a delivery run claims up to 500 pending events at once and attempts to deliver them. If delivery succeeds, the event gets marked "sent." If it fails, the event gets marked "failed" — and exactly one failed event must never stop the entire batch. That rule comes from the same lesson as the weekly digest mailing: a single problematic record must never take every other one down with it.

Step 4 — cleanup. Already-delivered events get removed after 30 days, so the outbox table doesn't grow without bound.

Who delivers today. Right now, the internal operations journal is the "subscriber" every event goes to — the same interface an external messaging system could later be plugged into, without anything about the storage or queueing part needing to change.

5. A measured example

In the live proof of the underlying transaction guarantee (see part 7): twenty loaded reviews produced twenty matching outbox entries. After a restart of the server process, both counts were still identical — no review without its event, no event without its review.

6. Limits of the method

• The pattern guarantees "delivered at least once," not "exactly once." If marking an event "sent" fails right after a successful delivery, the next run delivers it again. Every recipient on the other side therefore has to tolerate a possible duplicate delivery itself.
• There's always some delay between an event being stored and it actually being delivered — the pattern is built for "reliable, but not immediate," not for millisecond real-time delivery.
• The outbox table lives in the same database as the actual data — at very high event volume, it would eventually become its own scaling question. At today's scale, that isn't a concern.

7. Conclusion

The strongest property of this solution is that it introduces no new system to get a reliability guarantee — it reuses a guarantee the database already provides, for two things instead of one. That's the same underlying idea as part 7: don't build more technology, use the technology you already have in a way that makes the class of bug structurally impossible.

That closes out the second round of this series — two challenges instead of two algorithms, but committed to the same principle: ask the question precisely first, honestly weigh the alternatives considered, then build the solution that removes the class of bug instead of just hiding it.

← Part 7: The Bug That Hid Behind 227 Green Tests · Back to the overview