running-resumable-sync-jobs

name

running-resumable-sync-jobs

description

Design and review long-running batch or sync jobs that process many items against a remote API and persist checkpoint state — exit codes that report partial failure, checkpoints that are safe to resume, per-item tolerance vs. fatal abort, telling "zero" apart from "couldn't fetch", bounded retries, and honest dry-runs. Use when writing or reviewing a mirror/sync command, a nightly cron job, an importer, a paginated fetcher, or a background worker.

Running Resumable Sync Jobs

A sync job is a loop over N items against a service that will fail partway through. Everything hard about these jobs is what happens on item K of N: what the process reports, what it wrote down, and what the next run does with that. Get the happy path right and you still ship a job that silently under-collects, double-applies, or bills for work it never did.

The exit code is the only thing automation reads

The most common failure: per-item errors are printed as warnings, the loop continues, and the run finishes by saving state, committing, printing "Sync complete!" and returning success. A cron wrapper, a CI step, or a supervisor sees exit 0 and reports a healthy job forever.

# Bad — the warning goes to a log nobody reads; the process says "fine".
for user in users:
    try:
        mirror(user)
    except MirrorError as e:
        print(f"Warning: failed to mirror {user}: {e}")
        continue
save_state(); commit(); print("Sync complete!")
return 0

# Good — failures are counted and surfaced in the status the caller can act on.
failed = []
for user in users:
    try:
        mirror(user)
    except MirrorError as e:
        log.error("failed to mirror %s: %s", user, e)
        failed.append(user)
save_state()           # keep the work that succeeded
if failed:
    log.error("%d of %d targets failed: %s", len(failed), len(users), failed)
    return 1           # or 2, if you want "partial" distinguishable from "total"
return 0

Continuing past a failure is usually right — you want the other 99 items. What is never right is continuing and claiming success. If partial success must be acceptable to the caller, make it an explicit opt-in (--allow-partial), not the default silence.

Checkpoint what actually happened, not what you attempted

Resumable jobs record "item X is done" so the next run skips it. Two ways that record goes wrong, and both are worse than no checkpoint at all:

Attempted-but-unfinished recorded as done. State is saved for everything processed before the failure, so the next run skips those and the post-failure items stay pending forever — and because the run exited 0, nothing ever says so. This is the exit-code bug above compounding into permanent data loss.

Finished-but-not-recorded. A batch applies its units one at a time (N commits, N rows, N API calls) and the helper returns only an error when unit K fails — no count of the K−1 that landed. The caller leaves that item's state untouched, so the next run recomputes the whole delta and re-applies the units that already succeeded. Every retry over-applies.

# Bad — the caller learns nothing about how far the batch got.
def apply_batch(units) -> None:
    for u in units:
        apply(u)            # raises on failure; K-1 side effects already landed

# Good — report progress alongside the failure so state can advance.
def apply_batch(units) -> int:
    """Returns the number of units successfully applied."""
    applied = 0
    for u in units:
        try:
            apply(u)
        except Exception:
            raise PartialBatch(applied) from None
        applied += 1
    return applied

Either report the partial count and check it in, or make the unit of work atomic (one transaction, one commit) so "some of it happened" is not a reachable state. Pick one deliberately; the default — raise and discard the count — is the bug.

Per-item tolerance must be decided before the shared error handler

Jobs accumulate a shared handle_api_error(response) helper. These are almost always fatal by default: unknown status → log and exit. That is the right default for a single-target command and the wrong one for a loop over 900 items, where one deleted account should skip one item.

The trap is that the skip branch looks present but is dead:

# Bad — handle_api_error raises SystemExit on any non-200, so the skip below
# is unreachable. One 404 aborts the run after 800 successful calls and
# writes no output at all.
r = get(f"/users/{login}")
if r.status_code != 200:
    signal = handle_api_error(r)
    if signal == "retry":
        continue
    log.warning("Skipping user %s due to error", login)   # never runs
    skipped += 1
    continue

Branch on the cases you tolerate before delegating:

if r.status_code != 200:
    if is_rate_limited(r):
        handle_api_error(r)      # shared wait/retry policy
        continue
    log.warning("Skipping %s: HTTP %s", login, r.status_code)
    skipped += 1
    continue

Then report skipped in the summary. Two more rules that fall out of this:

• Grep for other callers when you change a shared handler's contract. A helper that is fatal-by-default with one tolerant caller is a helper whose other callers are all one bad row away from discarding a full run.
• Write output incrementally, or before enrichment. A job that fetches a list, enriches every element, then writes one file at the end loses everything to a failure in the last element. Persist the cheap list first.

Distinguish "zero" from "couldn't fetch"

Aggregators fold a failed fetch into the output as a plausible number. This is the quietest bug in the whole category, because the artifact looks complete.

# Bad — a rate-limited or transient clones fetch is written out as a confident 0
# and flows into the printed total.
row = {"views": views["count"], "clones": clones.get("count", 0) if clones else 0}

# Good — unknown stays unknown, and the run says how much it lost.
if clones is None:
    incomplete.append((repo, "clones"))
    row["clones"] = None          # or omit the row entirely

Related: a run that treats only some failures as decisive is inconsistent by construction. If a None from fetcher A increments skipped and continues, but a None from fetchers B and C is silently coerced to 0 and dropped from the aggregate, the trailing "skipped: M" line under-reports and the totals are wrong in a direction no reader can detect.

Cross-check totals against detail when the source gives you both. If a page publishes "1,759 contributions in 2024" and your parser extracts zero rows, that is a parse failure, not an empty year. A positive summary with an empty detail set must raise; zero-or-absent summary with an empty detail set is a valid empty result.

Bound every retry, and count per unit of work

A continue on a "retry" signal, inside a pagination loop, with no counter, re-requests the same page forever against a persistent 403 or 429. The job does not crash — it hangs, which no timeout-free supervisor will ever notice.

MAX_RETRIES = 5

retries = 0                       # per page, not per run
while True:
    r = get(url, params={"page": page})
    if is_rate_limited(r):
        if retries >= MAX_RETRIES:
            return items, False   # partial + incomplete flag; warn loudly
        wait_for_reset(r)
        retries += 1
        continue
    retries = 0                   # reset after any success
    ...

Counting per unit of work and resetting after a success is what lets a long multi-page fetch ride out repeated rate limits while still bounding retries of a single stuck page. A per-run counter kills legitimate long jobs; no counter hangs forever.

Write down whose job the waiting is. If the shared handler already slept until the reset timestamp before returning "retry", a caller that adds its own sleep(60) double-waits — a bug that is invisible in output and only shows up as a job that takes twice as long as it should. Document the contract once ("the wait already happened; re-request now") and hold every new caller to it.

Incremental cursors: re-include the boundary

A job that stores last_sync as a timestamp but compares at day granularity loses the boundary day permanently:

# Bad — skips days <= since. Anything that happened later on the same calendar
# day as the last sync is never re-fetched, and that day's count freezes.
if day_date <= since:
    continue

# Good — truncate the cursor to the comparison's granularity and re-include the
# boundary. Merging is keyed by date, so re-processing that day is idempotent.
if day_date < since.date():
    continue

The general rule: an incremental cursor must be exclusive only at the exact granularity it is stored at. If your merge step is keyed and idempotent — and it should be — re-processing the boundary unit costs nothing and closes the hole.

Dry-run must perform every in-memory mutation the real run does

Gate persistence and external side effects. Never gate the in-memory state changes the preview loop reads, or the preview describes a different job than the one that will run.

# Bad — under --dry-run the counters are never cleared, so the preview computes
# incremental deltas and reports "nothing to do", while a real --rebuild would
# re-apply the entire history.
if rebuild and not dry_run:
    state.clear_all_progress()

# Good — clear in memory always; only the write is gated.
if rebuild:
    state.clear_all_progress()
...
if not dry_run:
    save_state(state)
    rewrite_history()

Also make destructive scope explicit in the preview. A --rebuild that recreates history by removing every tracked file is correct only in a dedicated single-purpose repo; run it where other content lives and it deletes that too. Say what will be removed, not just what will be added.

Testing these paths

• Assert the exit code, not just stdout. A test that only checks for a warning string passes on the version that exits 0.
• Record sleep durations, don't stub them away. monkeypatch.setattr(mod, "sleep", sleeps.append) lets you assert the exact sequence (assert sleeps == [reset_wait, 0.1]), which is what catches a redundant extra wait. A no-op lambda _s: None cannot.
• Make the error persistent when testing a retry cap: serve a reusable error response and assert the request count for the stuck page. Also match on the response body if that is what your handler branches on — status alone may not select the branch you think.
• A missing cap makes the suite hang, not fail. Run the mutated suite under an external timeout; no output within N seconds is the reproduction.
• Test the resume, not just the run. Fail item K, then run again against the same state and assert the remaining items are attempted and the completed ones are not re-applied. A single-run test cannot see either checkpoint bug.

Checklist

• Non-zero exit (or an explicit --allow-partial) when any item failed
• Failure count and the failing identifiers appear in the final summary
• Checkpoints record only completed work; partial batches report their count
• Per-item tolerance branches before the shared, fatal-by-default handler
• Skip/dead branches verified reachable (grep the handler's other callers)
• Output written incrementally, or the cheap pass persisted before enrichment
• Failed fetches are None/absent in the artifact, never a coerced 0
• Summary-vs-detail contradiction raises instead of returning empty
• Every retry loop is bounded, counted per unit, and reset after a success
• The "who waits" contract is documented and no caller double-sleeps
• Incremental cursor re-includes the boundary unit; merge is idempotent
• Dry-run mutates in-memory state identically; only writes are gated
• Tests assert exit codes, sleep sequences, request counts, and a resumed run

Related: no-silent-pass for making in-process errors visible at the call site and for proving a regression test can actually fail.