The incident
An automation wrote the wrong value into an external system. Vendor A's order number went in as the tracking number for vendor B's order, and the customer could no longer track their shipment.
First cause — a single matching key
Matching was done on vendor name alone. With more than one open item for that vendor, it attaches to an arbitrary one.
Second cause — this is the real one
The regex written to *detect* bad values had ^\d{10,14}$ registered as a "valid format".
That is exactly the shape of this incident — a 10-digit order number sitting in the tracking field.
★★ The signature of the defect was on the validator's allow-list.
A rule added because "that's a normal format" let precisely that accident through.
★★ It could not be undone
At the time this was judged to mean "there is no correction API" — every guessed path (/invoices/correction, /invoices/update) returned 404.
That judgement was wrong. A correction path existed under a different name: the write is .../invoices, the correction is .../updateInvoices. The assumption was that a verb would hang off the same noun; it was a separate verb path. One search found it, and every affected record was fixed.
★★ A 404 on a URL you guessed means "I don't know the path", not "the capability doesn't exist". → see the dedicated entry below
⚠️ Being recoverable does not make the incident lighter. The wrong value was genuinely shown to the customer; the fix did not undo that, it covered it.
★ For irreversible writes there is no defence but validation beforehand. Do not design as though a corrective path will exist.
Rules
Aside — a defect that was harmless because it never landed
This defect had already been documented weeks earlier. At the time the write never actually reached the downstream system, so there was no damage, and it was filed as "observed".
★★ A defect that was harmless because it never landed has not been fixed. It fires unchanged on the day the path opens.
The original incident's four causes — every one was a plausible-looking value passing
| Cause | Fix |
|---|---|
Tracking regex (\d{10,14}) mistook an order number for a tracking number | Drop the pure-digit pattern; accept carrier-prefixed patterns only |
Substring shipped in "getting your order ready to be shipped" judged as dispatched | A separate "not yet" pattern; split order-received from shipped |
| An empty keyword passed every filter and wrote into the first order | No keyword → abort |
| Two or more orders from one vendor → wrote into the first one | Return AMBIG, stop auto-entry |
★ A fix written in the notes and a fix applied at every call site are different facts. When it recurred, ① and ④ turned out to be missing from that path. Count every place that uses the value, not the function you fixed.
The mirror image — a correct property used as a violation metric is a false-positive generator
Above, the defect was on the allow-list. This is the opposite. A checker for three tables used "scatter of text start-x" as its violation metric and reported all three as "alignment broken". In the right-aligned amount column the right edge was identical on every row; the scattered start was nothing but differing digit counts. In a centered column, badges of different widths scatter at both ends — that is the definition of centered. The disproof: after the "fix", not one measured value changed. The real culprit was a different column.
| Alignment | Must match | May scatter |
|---|---|---|
| Left | start (left) | end |
| Right | end (right) | start |
| Center | center | both ends |
★ Before choosing a metric, answer "if this number is large, what is wrong?" Without an answer, the number manufactures a misreading that looks like evidence — worse than a bare guess, because it has a figure attached.