The incident
Cron entries grew to 74 and crossed the process limit.
fork: Resource temporarily unavailable★★ This is where it gets bad. Every external command needs to fork, so ps, pkill, cat, chmod and crontab all stop working. You cannot see what is running, so you cannot tell what to kill.
★ The commands you normally check with become unusable exactly when you need them. Same shape as a full disk refusing new SSH sessions — exhaustion takes observability first.
★★ An emergency route that consumes the normal resource is unavailable in an emergency
/proc is container-isolated and shows only your own; you cannot kill anything elseThe one remaining route was deleting entries from the cron list screen in the control panel — the only path that consumed no process.
★ When designing a recovery route, look at what resource that route consumes. If it is the same resource that is exhausted, it is not a recovery route.
★★ Recovery technique — what runs without forking
Shell builtins (echo, read, kill, for) run without forking. And once a single interpreter is up, os.kill, file writes and subprocesses inside it can do everything else.
★ Near zero resources, switch from "issue many commands" to "one execution that does all of it." The whole problem is securing one process.
Removing the cause does not remove the symptom
★ Deleting the cron entries does not kill the processes already spawned. You have to wait for recovery. Concluding "it isn't fixed" and intervening further makes it worse.
★★ Count schedules by occurrences per hour, not by lines
Half the problem was ten sub-hourly cron entries.
1-59/4 · 3-59/8 · 1-59/5 · 2-59/9 · 9-59/10 · ...Each reads as "once every few minutes" and looks small. Ten of them together spawn close to 100 per hour.
★ Counting cron entries is not enough. Sum the per-hour rate of each line.
★★ And there is a second axis — how many land in the same minute
The other half was 17 entries scheduled on the hour (minute 0). All 17 spawn simultaneously, every hour. That instant blew past the process limit and locked out remote access too.
★★ The same total blows up when it clusters. 17 runs per hour is nothing when spread out; land them in one minute and that minute exceeds the limit. The limit applies to instantaneous concurrency, not to the hourly average.
★ Equal periods mean equal phase. Seventeen lines of 0 * * * * always fire together. Nobody intended to bunch them — minute 0 is the reflex default, so they bunch themselves.
★★ Read a schedule three ways — line count, occurrences per hour, and concurrent occurrences in the same minute. The third one is what actually hits the limit.
→ Scatter the minute. Use 3, 7, 11, 17 … instead of 0.
Fix — a sequential runner
Keep the work list in a file and have one cron entry pull one item per tick.
pgrep approach matches itself — don't)timeout shorter than the interval so ticks never overlap★★ Availability beats throughput. One sequential worker beats twenty concurrent ones.
★★ Raising the gate looks like more throughput, but crossing the limit stops everything. Aiming for 20 lands you at 0.
★★★ How it got there
Batch size was raised, the concurrency gate was raised, and cron entries were added. All three at once. Each was a reasonable-looking throughput adjustment on its own.
★ High priority does not mean it is safe to do a lot at once.
★ On a capped resource, the optimum is in the middle, not at either end. Elsewhere the same system had a limit set far too low and was suppressing itself; here one was set far too high and everything stopped. Two ends of the same mistake.