The Bug That Showed Up Three Times Wearing a Different Disguise

Read, check, write looks completely safe. Under real traffic it can quietly corrupt your data. Claude got it wrong three separate times before I stepped in and taught him why.

Here is a piece of code Claude wrote for me. It looks correct, it passed every test we had, and it is still wrong.

var row = await db.Things.FindAsync(id, ct);   // read
if (row.Version >= incoming) return;           // check
row.Version = incoming;                         // modify
await db.SaveChangesAsync(ct);                   // write

Read the row, check whether the incoming change is newer, and if it is, write it. Sequentially it behaves exactly as you would expect. Every unit test passes. Nothing in a quick read flags it either, unless you already know the shape of the bug.

Then picture it under real traffic, which is where I knew to look. It starts producing states that should have been impossible. A value that was supposed to only ever move forward would occasionally jump backward. The reason is the gap between the check and the write. Two requests arrive at nearly the same moment, each in its own database context. Both read the old row. Both pass the check, because at the moment they checked, the other one had not written yet. Both write. The last one wins, and an older event can stomp on a newer one. It is a classic time-of-check to time-of-use race, and nothing in that code holds a lock across the gap.

What made it memorable is that Claude did not make this mistake once. It made the same one three times, in three different disguises, across three features, and each time the code looked perfectly reasonable on its own.

The same mistake, three costumes

The first time, it was a counter guarding how many times something could be fetched. Under load testing it sailed past its own limit, because two requests both read the same count and both incremented from it.

The second time, it was a subscription webhook overwriting an entitlement row. A retried or out-of-order delivery flipped a test subscription to expired, because “apply whatever just arrived” assumed the events came in order. They do not.

The third time, it was refresh-token rotation. Two requests presented the same one-time token, both rotated it, and the system minted two valid token families from a token that was supposed to be used exactly once. That one is worse than a wrong number in a database, because it quietly defeats the reuse detection that the whole rotation scheme exists to provide.

Three features, the same root cause each time. The thing I kept flagging in review was not about counters or webhooks or tokens. It was that a check sitting in application code, separated from the write by even a few milliseconds, is not a guard at all when more than one caller can reach the row.

The fix is to let the database do the checking

The fix I gave it is to stop checking in C# and push the condition into the write itself, so the comparison and the update are a single statement the database serializes on the row lock.

var claimed = await db.Things
    .Where(t => t.Id == id && (t.Version == null || t.Version < incoming))
    .ExecuteUpdateAsync(s => s
        .SetProperty(t => t.Version, incoming)
        .SetProperty(t => t.State, next), ct);
if (claimed == 0) return;   // someone newer already won, or this is stale

The number of rows affected tells you whether you won. If it comes back zero, someone newer got there first, or the event was a duplicate, and you simply do nothing. There is no window for two callers to both believe they are the winner, because the database evaluates the condition and the write as one atomic operation.

For the webhook specifically, this pairs with a second habit: carry the provider’s own event timestamp, apply an event only if it is newer than the last one you applied, and return a 200 even when you decide to ignore a stale duplicate so the provider stops retrying. Webhooks are at-least-once and unordered. Plan for both.

The part that bites quietly: you cannot test this with a sequential test

The reason this keeps getting past initial tests is that a normal unit test runs one request at a time, and one request at a time never reproduces the race. You have to fire concurrent requests on purpose, with something like Task.WhenAll over real HTTP calls, and watch the impossible state appear.

There is a sharp edge in the testing itself. Entity Framework’s InMemory provider cannot run ExecuteUpdate at all, so those tests have to move to SQLite in memory. And SQLite will not translate a DateTimeOffset comparison inside the update predicate, while Postgres will, so keep immutable conditions like an expiry as a separate pre-read check and put only the genuinely contended state in the atomic WHERE. The most reliable proof is a real Postgres integration test that actually races the row.

If I had to compress the whole thing into one sentence: a test that only ever runs one thing at a time can give you a green check mark on code that falls apart the moment two things happen at once. Concurrency bugs do not live in the code you read. They live in the timing you forgot to test, which is exactly why fast, confident, AI-written code still needs a reviewer who has seen this one before.