name: jcstream-sweep-debugger
description: Use when investigating why a JCStream sweep didn't produce fresh data — the roster count is flat, the changelog is empty, the site rendered stale photos, or the GitHub Actions sweep workflow showed exit-0 but no commit. Covers the silent degraded-roster fallback in scraper/sweep.py, the health thresholds in sweep_guards.py, and how to read changelog.json / history.json / current.json together. Trigger phrases: "sweep didn't update", "stuck count", "no changes in changelog", "scraper looks idle", "stale photos", "partial sweep", "sweep bailed", "sweep timed out", "nameless records", "detail watchdog".
JCStream sweep debugger
You diagnose sweep failures. The most common failure mode is silent: sweep_looks_healthy (in scraper/sweep_guards.py; the underscored name in scraper/sweep.py is a back-compat alias) returned False, sweep.py kept the last-good data/current.json, and the workflow exited 0 with no commit. The site looks fine but is stale.
The triage flow
1. Did the sweep workflow run at all?
Check GitHub Actions for the most recent sweep run:
git log --oneline -20 | grep "data+site: sweep"
Gaps in the timeline (>30 min) mean the cron didn't fire — that's a GH Actions / repo-permissions problem, not a scraper problem.
2. Did the scraper bail on the health check?
The workflow log will show the sweep's sweep_looks_healthy decision. Without log access, infer from data:
- Compare today's
data/current.json inmate_count to yesterday's: if it's identical for >2 cycles, the gate likely tripped.
- Tail
data/changelog.json: if the most recent timestamp_utc is >1 hour old, no new events.
3a. Which list-side threshold tripped?
From scraper/sweep_guards.py:
| Threshold | Meaning |
|---|
SWEEP_MAX_FAILED_FRACTION = 0.10 | >10% of surname fetches errored → bail |
SWEEP_MIN_ROSTER_FRACTION = 0.5 | new roster <50% of last cycle → bail |
SWEEP_BOOTSTRAP_FLOOR = 50 | first-ever sweep needs ≥50 inmates → bail |
To distinguish:
- High error rate → HCSO rate-limited the scraper. Look at the workflow log for HTTP 429 / generic-block-page hits. The fix is usually time (wait an hour) — not a code change.
- Roster collapse → HCSO published a degraded list (frequent during a system migration). Check
https://www.hcso.org/justice-center-services/inmate-search/ manually.
- Bootstrap floor → a brand-new deploy with
data/current.json missing. Run a manual sweep with workflow_dispatch.
3b. Did the detail-page watchdog trip?
Even when the list sweep stays green, check_detail_watchdog (scraper/sweep_guards.py, called at scraper/sweep.py) can refuse the cycle if the per-inmate detail pages stop yielding names or photos. This is the path that fires on "stale photos but fresh roster count" / "nameless inmates".
From scraper/sweep_guards.py:
| Threshold | Tier | Meaning |
|---|
DETAIL_WATCHDOG_MIN_SAMPLE = 10 | gate | below this many detail attempts the watchdog is silent (small samples are noisy) |
DETAIL_WATCHDOG_NAME_FLOOR = 0.70 | WARN | <70% of detail attempts parsed a name → log warning, still write |
DETAIL_WATCHDOG_PHOTO_FLOOR = 0.50 | WARN | <50% of detail attempts yielded a photo → log warning, still write |
DETAIL_WATCHDOG_BLOCK_MIN_SAMPLE = 100 + DETAIL_WATCHDOG_BLOCK_NAME_FLOOR = 0.60 | BLOCK | both must hold (≥100 attempts AND <60% named) → roster_ok=False, keep last-good |
A WARN-tier trip leaves the roster fresh but the warnings in the workflow log are the early signal that HCSO's detail-page HTML is shifting (parser regression imminent). A BLOCK-tier trip looks identical to a list-side bail from the outside — exit 0, no commit.
3c. Other silent-bail / partial-write paths in scraper/sweep.py
| Path | Where | Symptom |
|---|
| Wall-clock cap | SWEEP_WALLCLOCK_HARD_CAP_S = 22 * 60 at scraper/sweep.py, enforced at scraper/sweep.py | sweep finishes after ~22 min with a partial roster persisted; diff/changelog are skipped (clean_finish=False), so the count moves but no events appear |
| Checkpoint guard | scraper/sweep.py | mid-sweep save_current checkpoints are skipped when in-memory roster is <50% of previous; a stuck count can come from this even though the final-write path looks fine — log line: "checkpoint skipped at N/M details" |
| Corrupt-snapshot bail | scraper/sweep.py (SnapshotCorruptError) | returns 0 immediately; the broken data/current.json is left in place for inspection — log line: "refusing sweep: data/current.json is unreadable" |
save_current failure | scraper/sweep.py (OSError) | disk-full / atomic-rename failure: snapshot unchanged, changelog and prune both skipped — log line: "save_current failed (...); skipping changelog and prune" |
| Photo prune skip | scraper/sweep_guards.py (PHOTO_PRUNE_MAX_FRACTION = 0.5) | when >50% of stored photos would be deleted in one cycle, prune is skipped wholesale — symptom is "photos for released inmates aren't disappearing" — log line: "photo prune would remove N/M photos (>50%) - skipping prune" |
Atomic write contract: data/current.json is written via tmp + os.replace in scraper/store.py (_atomic_write_text), so a half-written snapshot is never published — a stale current.json is always intact, never truncated.
4. Cross-check the data files
| File | What it tells you |
|---|
data/current.json | Latest accepted roster snapshot. generated_utc is when the snapshot was written (not when the sweep ran — those diverge during a bailed sweep). |
data/changelog.json | Append-only log of booked/released/updated events. Length grows on every accepted sweep. |
data/history.json | Daily roster-size + churn counts. Stubby today (≈200 bytes). Owned by web/build.py (_update_history), not by the sweep — a stale history.json points at the build/Pages workflow, not the scraper. |
| Workflow logs (Actions) | Per-surname HTTP status, error count, gate decision. |
If current.json.generated_utc is recent but changelog.json is unchanged → the sweep was accepted but no events flipped (genuinely quiet day) or the sweep tripped the wall-clock cap and the diff was skipped (check the log for "wall-clock cap reached").
If current.json.generated_utc is stale and changelog.json is stale → the gate has been tripping. Read the recent workflow logs.
When to file a code fix
- Threshold is too tight for the actual error envelope (HCSO consistently returns 8% errors and the floor is 10%) → tune
SWEEP_MAX_FAILED_FRACTION in sweep_guards.py with telemetry to back it up.
- A new HCSO HTML quirk is causing parsing failures → fix the parser, don't relax the gate.
- The bootstrap floor is biting a legitimate restart → drop a known-good
data/current.json into place rather than weakening the check.
When NOT to file a code fix
- HCSO is rate-limiting → wait it out, this is normal.
- HCSO is publishing a degraded list → the silent fallback is protecting the site; let it.
Anti-patterns
- Lowering
SWEEP_MIN_ROSTER_FRACTION to make a bad sweep go through.
- Editing
data/current.json by hand.
- Re-running the workflow without diagnosing — you'll get the same answer.
Verify a fix
python -m pytest -q tests/test_sweep.py
