Symptom
A scheduler setting moved from 120 runs/hour to 88 and back to 120 within a day. Nobody reverted anything. Two sessions each reported having "cleaned it up."
Cause
Sessions cannot see each other. Each read the current state, improved it by its own criterion, and saved. One was optimizing throughput; the other was optimizing load.
★★★ Both were right by their own measure. Two optimizers with different objectives sharing one variable do not converge — they oscillate.
★★★ And oscillation does not look like a failure. Every individual value is exactly what someone intended, and the logs only ever say "adjusted." The only way it surfaces is plotting the value over time.
★★★ Most shared resources do not detect conflicts
Work with git long enough and you start expecting that concurrent edits collide. That is git being unusual.
| Concurrent edit | |
|---|---|
| A git repo | rejected, rebase required |
| Scheduler config, config files, DB rows, an external console | ★ last-write-wins. Silently overwritten |
★★ Where there is no detection, convention is the only defense.
Fix
★ "I set it to 88 last time" is not evidence. Another session may have run since.
⚠️ ★★★ That shared state file is not authoritative either
A snapshot regenerated on a schedule (say, every two hours) is stale by up to that interval. Changes inside the window are invisible.
★★★ Treat the snapshot as the source of truth and you reproduce "I read it, I changed it, and it still oscillated." → The shared file tells you what to be careful about; the current value gets read from the live resource immediately before you change it. Put both jobs in one file and the stale number takes the canonical slot.
★ Stamp the generation time at the top and make readers check the age. An undated snapshot always reads as current.
⚠️ ★★ And a convention only works if it is followed
"Read this at session start" is behavior outside the code. Skip it and nothing happens.
★★★ Making it impossible to change without reading is easier than getting people to read. → Wrap the mutation path in a script that prints the current value and the snapshot's age first. Turn the convention into a procedure and you no longer have to verify it was honored.
⚠️ ★★★ Follow-up — the "do not touch" list was broken the same day
Point 3 above (keep a hands-off list) was written, and within a day we measured it. Five of the six protected entries had been changed by another session.
★★★ What matters more is that every change went the same direction. All five had their intervals lengthened (slowed) — because the other session was cutting toward a numeric "runs per hour" ceiling.
★★★ The objective lived in the code as a number; the protection list lived in a document as a sentence. The number wins. Unless the protection list is an input to the reduction logic, the list might as well not exist.
★★★ And cutting by frequency can be inversely correlated with importance
The two cut hardest were the notification loop and the runaway-prevention guard. Both ran frequently because they had a reason to.
★★★ Frequency measures cost, not importance — and a reduction pass only looks at cost. → Rank cut candidates by frequency ÷ importance, not frequency.
★★★ Safeguards get cut first, specifically. A safeguard looks like cost, and the incidents it prevented never happened, so they don't look like benefit. This is one step beyond this collection's a safeguard creates the next trap family — this time the safeguard switched itself off.
★★★ The ladder of defenses — each rung needs less cooperation than the last
| Rung | Requires | Result |
|---|---|---|
| Write it in a document | the other party reads it | ❌ 5 of 6 violated |
| Enforce via a gateway | the other party uses the gateway | bypassable |
| Auto-restore | ★ nothing at all | reliable |
★★★ Making it impossible to change without reading beats getting them to read — and restoring it when it changes beats both.
⚠️ ★★★ Auto-restore has three traps of its own
★ And check who receives the restore alert. If it only reaches a human, the session that caused it never sees it. Write it to the shared state file too, so the next session reads it.
How to verify
★ Keep a time series of the values you change. Oscillation is invisible in any single reading and obvious in the trend. A value bouncing between two points is not tuning — it is two parties fighting.
★★ And periodically measure whether the protection list is actually being honored. Creating a list and having it work are different facts — we found out only after 5 of 6 had been broken.
★★★ Follow-up — that violation was not a discipline problem, it was a symptom of an impossible target
Feeding the protection list into the reduction logic revealed it immediately. The protected entries alone already consumed 80% of the ceiling. Honoring the ceiling *and* the list was impossible from the start.
★★★ The other session did not violate the rule because it failed to read it. Even having read it, hitting the target required violating it.
★★★ Give an impossible target and a constraint breaks. And the broken constraint is the visible part; the impossible target is not. While everyone asked "why wasn't the rule followed?", nobody asked "was that target even reachable?"
★★ Two constraints contradicted each other and had never been checked together. The ceiling lived in an ops document, the protection list in a config file — kept apart, the contradiction is invisible. Each was reasonable on its own. → ★★★ When you add a constraint, verify it is simultaneously satisfiable with the existing ones.
★ Which is why auto-restore alone was not enough. Restoring reverts the symptom and does nothing about an impossible target. What actually fixed it was re-deriving the ceiling, not the restore mechanism. → ★★ When you see oscillation, ask "are both objectives simultaneously achievable?" before "who broke the rule?"
★★★ And the metric the ceiling was set on turned out to be a proxy
Re-dissecting the incident, the cause was not call frequency alone but frequency × batch size — concurrent execution count. Low frequency with a large batch blows up just the same.
★★★ Cap a proxy metric and you get both failure modes at once — you miss what you should have stopped, and you stop what you shouldn't have. Here it let the large batch through and blocked the protection list. Same root cause.
★★ A proxy gets chosen because it is easy to measure, not because it is right. → When you gain the means to measure the real thing, move the cap onto it. We had the guard log peak concurrency and loosened the frequency cap.
★★★ Finally — "impossible" turned out to be "we weren't doing it"
The premise under all of this was "sessions have no way to share state." False. The shared store was already on the server and every session could read it. What blocked it was not the medium but the write schedule — each session committed once, at the end, so work in progress was invisible to everyone.
★★★ Latency in a shared medium is a property of your write policy, not of the medium. Having the state file written and pushed periodically turned "asynchronous, two hours stale" into "near real time."
★★ This collection already contains an entry where our own default was read as a platform constraint. Same shape — what we believed was a constraint was our own habit. ★ Before writing down "that's impossible," ask "have we ever actually tried?"
★★★ Over half a year this shape has appeared four times — our pipeline's default read as a platform ceiling, our estimated cost read as a fixed cost, a 404 read as a missing feature, and "state sharing is impossible" read as a limit of the medium. Three of the four were ours and one was theirs, and all four felt identical.
★★★ Without a recorded provenance, your assumption and their constraint are indistinguishable. → Write the source next to every constraint you record — a platform doc URL, a constant in our code, or our own estimate. Put all three in the same kind of sentence and you lose the ability to tell them apart later.
★★ Finally, attach the derivation to every derived value
Setting the new limits, batch size landed at 600. That 600 comes from a 280-second task timeout — a larger batch cannot finish anyway.
★★★ Whoever changes that timeout will not know 600 needs changing. When one parameter determines the meaning of several others and the link is written down nowhere, the derived values do not move with the original. → Put it beside the value: 600 = derived from timeout 280s. ★ Same rule as this collection's store the population with the baseline — the number alone is not enough; where it came from is part of the value.
★ And that timeout has a record. The same value previously caused cleanup traps to be skipped on kill. The more places hang off a value, the more it sits somewhere easy to change.