Background
After switching to "hand the work to the background and answer immediately," call success rates guaranteed nothing. So monitoring was set on "was the log file updated within N minutes?"
Trap ① — the watched log was written only conditionally
The first choice was the job log. But that log is written only when there is something to process. Zero new items and it writes nothing at all. → Quiet periods got reported as outages.
★★★ A log that records "something happened" and a log that records "it ran" are different things. Monitor the second using the first and you get alerts precisely when everything is fine.
★★ And repeated false alarms get the alert turned off. The real silence that arrives afterwards goes unseen — the same place as this collection's repeated notifications make notifications meaningless.
→ ★ A heartbeat writes one line on entry, unconditionally, regardless of the work's outcome. Monitor that line.
Trap ② — ★★★ but a heartbeat only catches silence
If the log is written unconditionally, it is written when the job fails too. ★★★ You know "it ran"; you do not know "it succeeded." A backgrounded job dying quietly is still invisible.
→ Split it into two layers.
| Layer | Watches | Catches |
|---|---|---|
| ① Heartbeat | entry log's update time | invocation stopped |
| ② Completion record | what the job writes at the end | the job failed or was cut short |
★★ Without ②, "invoked exactly every ten minutes and accomplishing nothing" looks perfectly healthy.
★ Record item counts in the completion record. But ⚠️ do not turn a run of zeros straight into an alert — that is trap ① again. There are hours where zero is correct. Read it as a trend, not an alarm.
★★ The monitor must live on a different resource than what it monitors
Moving scheduling to an external service flips the failure mode to silence. When you call out, failure arrives as an error; when something external calls you, failure arrives as nothing at all.
★★★ Move a resource outside and move its monitoring with it — let the outside watch what you gave to the outside and both die together. Keep a minimal dead-man switch inside. ★ Same principle as this collection's an emergency measure that consumes normal resources is unavailable in an emergency.