A synchronous bulk route dies at the caller's timeout

Symptom

Processed a high-count job synchronously inside one HTTP request; no response comes back. Partial work was committed, but with no response you can't tell how far it got (retrying risks duplicates).

There is more than one timeout. This is the point.

Stepping on this mine four times, each time a tighter timeout appeared.

Axis	Nature
Worker timeout	Mine. I can raise it.
External scheduler timeout	Can't raise it. Platform hard ceiling.
Worst-case latency per item	External dependency, no guarantee

★ A count cap (limit) is not a time cap. limit=10 at a worst case of 8s per item is 80s. Under the worker timeout, over the external scheduler's 30s.

The intermediate fix and its limits (worth keeping)

First attempt: a wall-clock budget. Check elapsed time *before* starting each item; on hitting the budget, stop and return {processed, remaining, budget_exhausted} as 200. The next call picks up the rest. Grab, process, and commit one item at a time, so zero half-done residue on interruption.

Still not enough. Two reasons.

1. Even with a 22s budget, the one item in flight can overrun. Against a hard ceiling, "usually fits" is meaningless.
2. The route stacked two heavy jobs. The budget was only on the first; the second ran unbudgeted for another 20–40s. Worse, the second job starved and never ran at all. The external scheduler's red doesn't tell you which hop starved.

The final fix: respond immediately, don't count backwards from a budget

• The route returns 202 immediately and hands the real work to a background thread
• Scheduler verdict = immediate 202 = success. Real results are read from logs.
• A re-entry guard. But idempotency is the foundation of multi-worker safety; the lock is auxiliary.
• The thread budgets itself to exit. Keep the order:

job budget < scheduler interval < worker timeout

• Never stack two heavy external I/O jobs in one request. If unavoidable, reserve a minimum budget for each so neither starves.

Result

http=202 · 0.24s. Immediate-response design makes it constant time regardless of queue length. With 100× headroom against the hard ceiling, that axis drops off the worry list.

Rule

When the caller's timeout is a platform hard ceiling and the job might not finish inside it, don't shave the budget. Go immediate response + background + idempotent resume + re-entry guard. Budget-shaving only works when the ceiling can be raised.

Don't build a separate state store

Use the work items themselves as state (draft / published / meta flags *are* the progress). Stateless resume with no job store. If the work is already idempotent and resumable, an orphaned background thread is fine: the next call picks it up.

Follow-up: measure the cost of doing nothing first

With zero remaining, a tick still took over 30s. The tick was fetching the full list twice, every time. Even with nothing to do, that fixed cost burns.

→ A completion cache that skips immediately after convergence, fetch the list once per tick and share it, and log "0 targets" as none (normal) vs fetch failed (abnormal).

★ Tune a periodic job only after removing its fixed cost. Then measure, then decide.

---

---