Symptom
*_done.json stores only {id: true}; what was written is gone. When bad output is discovered later, you can't count how many of the ~2,000 completed items are bad without hitting the API again.
Fix
Minimum three fields per done entry: {id: {ts, len, src}}. With the generated body length and the source, a full audit can be done locally.
★★ A prefix-format value changes which operator is correct
Putting a reason into the value is good — it records *why* something was handled that way.
done[x] = "ok:1234567890" / "skip:policy_excluded" / "manual:needs_review" / "fail2"Then the check must be startswith.
done_ok = str(v).startswith(("ok", "skip", "manual"))★ != "ok" and not in ("ok", "skip") pass for every prefixed value.
"ok:1234567890" != "ok" is True → every completed item gets reprocessed. No error, normal logs, and the queue never drains.
Safer shape — separate the verdict from the detail
done[x] = {"state": "ok", "detail": "1234567890"}Pack two things into one value and every reader has to honour the parsing convention; one place gets it wrong and it leaks.
The same operator hazard, other direction
| Value scheme | Wrong operator | Result |
|---|---|---|
"ok:123" | != "ok" | done read as not-done → infinite reprocessing |
draft / draft_pending | == "draft" | the second one is silently dropped |
★ When values share prefixes, both == and != are dangerous. One drops items, the other passes everything.
★ Decide the value format and the check code together. Change the format later and the check flips silently.
★★ Not saving before continue throws the decision away
If a loop changes state in memory and then continues, a crash or an interrupt before the next flush erases the decision itself. The next run sees the item as if for the first time.
★ Save the moment you change state. Do not rely on a save at the end of the loop.
This is one layer away from "record failures too" — there the record was never *conceived*; here it was made and then never written.
Four shapes produce the identical symptom
Four different ways to get the same progress file wrong. All four were hit in one day.
| Shape | Symptom | |
|---|---|---|
| 1 | Changed the value format, left the check | "ok:123" != "ok" passes → reprocess completed work |
| 2 | Compared with == | prefix-sharing values silently dropped |
| 3 | Marked ok from the response alone | no re-query → the whole completion log is fiction |
| 4 | Did not save before continuing | the decision vanishes → infinite reprocessing |
★ None of the four raises an error, and all four leave the queue full.
So if you fix one and the symptom stays, read it as "there is another shape", not "it is not fixed".
The rule — you need all three
startswith — with prefixed values, == and != are both wrong★ Miss any one and the queue stays full. And the symptom does not tell you which one you missed.
The trap you step on next: confusing "couldn't process" with "no longer needs processing"
Don't record failures in done and you get an infinite retry loop. Classify "no longer needs doing" as unprocessed and the list never empties; the batch repeats the same work every 20 minutes (385KB of logs and counting).
→ Record failures too, but as a distinct state from success.