poller-extend-readonly-audit

name	poller-extend-readonly-audit
description	Use when a scheduled poller/job currently advances a pipeline but never re-checks the artifacts it has already produced, and you need to add a read-only audit step that surfaces drift / structural problems / AI-taste regressions without mutating any files.

Poller: add a read-only audit step

Pattern for extending an existing pipeline poller (cron job) so it not only advances new work but also re-checks the artifacts it has already written. Critical constraint: audit is read-only — never rewrite source-of-truth files based on the audit alone.

When to use

• Existing cron/poller only processes "new batches" → 盲点: prior outputs are never re-validated
• User says things like "should check ALL the X files", "everything", "every batch" — signals full-coverage concern
• Output dir contains many files (1000s) but they were only scored at creation time
• Scorer logic may have changed since creation (drift possible)

Steps

1. Locate the config-driven output dir (don't assume hardcoded paths)

Many pipelines read output_dir from config/pipeline.yaml. Audit must scan the same path the writer writes to, otherwise it inspects the wrong collection.

grep -E "output_dir|OUTPUT_DIR" config/*.yaml
# Verify with: ls -la "$(yq '.pipeline.output_dir' config/pipeline.yaml)"

2. Independently ground-truth the count BEFORE writing audit code

Use find or pathlib.glob from a sandbox script to enumerate every <artifact> file. Record the exact count. The audit's audited=N number must match this count when you do the first dry-run — that's your smoke test. If your dry-run shows a wildly different number, you're scanning the wrong dir.

from pathlib import Path
root = Path('<output_dir>')
files = [d / '<artifact>' for d in root.iterdir() if d.is_dir() and (d / '<artifact>').exists()]
print(f"ground truth: {len(files)}")

3. Audit command shape

Pattern that worked on anti-ai-batch-poller (Python, batch_runner.py):

def cmd_audit(args, config, state):
    """Read-only audit of every <artifact> in the pipeline's output dir."""
    import time
    styles_dir = PROJECT_ROOT / config["pipeline"]["output_dir"]
    target = config["scoring"]["eight_dim_max"]
    t0 = time.time()

    # Four buckets — keep these distinct so cron reports stay grep-friendly
    structural = []   # (id, [issue_codes])
    drift = []        # (id, stored_score, cur_score, kind)
    ai_trace = []     # (id, dim, score)
    size_problems = []

    for sd in sorted(styles_dir.iterdir()):
        if not sd.is_dir():
            continue  # skip .DS_Store etc.
        h = sd / "<artifact>"
        m = sd / "metadata.json"
        if not h.exists():
            continue
        try:
            html = h.read_text(encoding="utf-8")
        except Exception as e:
            structural.append((sd.name, [f"read_error:{type(e).__name__}"]))
            continue

        # size checks (cheap, run first)
        if len(html) < 1024:
            size_problems.append((sd.name, len(html), "<1KB"))

        # structural checks — explicit issues list, not booleans
        issues = []
        low = html.lower()
        if "<style"   low: issues.append()
            low:    issues.append()
            low: issues.append()
            low:  issues.append()
         issues:
            structural.append((sd.name, issues))

        
        scores = score_html(html)
        cur_pass = scores[] <= target
         dim, val  scores.items():
             dim == : 
             val > :
                ai_trace.append((sd.name, dim, val))
         m.exists():
            :
                meta = json.loads(m.read_text())
                st = meta.get()
                stored = meta.get(, {}).get(, -)
                 stored >= :
                     st ==    cur_pass:
                        drift.append((sd.name, stored, scores[], ))
                     st ==   cur_pass:
                        drift.append((sd.name, stored, scores[], ))
             Exception:
                

    
    (
          )
    ()

Register --audit in argparse alongside the existing flags. Single-file change, no new modules.

4. Add to cron prompt — small, explicit, marked NEW

Steps:
1. cd <project>
2. source .venv/bin/activate
3. python -m <pkg>.pipeline.batch_runner --poll
4. python -m <pkg>.pipeline.batch_runner --consume
5. python -m <pkg>.pipeline.batch_runner --finalize
6. python -m <pkg>.pipeline.batch_runner --audit     # NEW: scan ALL artifacts
7. python -m <pkg>.pipeline.batch_runner --status

Report counts only — passed/failed per batch, plus AUDIT_SUMMARY line for the audit step.

5. Verify end-to-end

After updating the cron prompt, do not just rely on dry-run. Trigger one real tick:

hermes cron run <job_id>

Then read the latest output file in ~/.hermes/cron/output/<job_id>/ and confirm:

• The ## Response section contains the AUDIT_SUMMARY line
• The audited= number matches the ground-truth count from step 2
• The cron shows last_status: ok in hermes cron list

Pitfalls (verified 2026-06-28)

0. Command dependency chain (API key): --poll, --consume, --finalize all call get_api_key(config) → resolve_api_key() and will raise ValueError if the env var is unset. --audit and --status are fully standalone — they never touch the API. So the cron prompt's step ordering means steps 3-5 can fail while steps 6-7 still succeed. When reporting, always run all steps and report each independently — don't abort the whole cron tick on a missing key. Verified 2026-06-28 on anti-ai-style-factory: --poll/--consume/--finalize all failed with Missing API key: set OPENAI_API_KEY env var, while --audit (14,623 files, 67.68s) and --status (8 batches) completed successfully.

1. Wrong output dir: find . -name "reference.html" shows 15111; output_dir in config might point elsewhere (./styles). Always grep config for output_dir before writing audit code. The dry-run's audited=N IS your config-vs-reality smoke test.

2. Path.iterdir() includes .DS_Store and similar: top-level output/ or styles/ dir often contains macOS metadata files. is_dir() check already filters them, but be explicit: if not sd.is_dir(): continue and add a code comment.

3. Drift vs ai-trace overlap: a file with total > threshold and a non-zero dim will appear in BOTH drift and ai_trace. That's fine — they answer different questions ("did the verdict flip?" vs "what patterns are present?").

4. Large file counts slow the cron: 15096 files takes ~12s. Fine for 5-minute cadence. If you have 100k+ files, sample or parallelize.

5. Don't auto-fix from audit: audit reports problems but never rewrites. If user wants auto-regeneration of failed styles, that's a separate --repair step with explicit submit logic — separate decision because of infinite-loop risk.

6. Sandbox scripts crash on .DS_Store: iterdir() raises NotADirectoryError on macOS files at root. Always first.

Done criteria

• Cron prompt has the audit step
• Dry-run report's audited=N matches independent ground truth count
• hermes cron run <job_id> succeeds and output shows AUDIT_SUMMARY line
• last_status: ok in hermes cron list after the run

Companion repair step (when user says "fix everything")

The audit is read-only by design. Once it runs and surfaces a problem, a separate repair step is needed. Verified 2026-06-17 on styles/*/reference.html (14.6k files):

Repair classification (do NOT skip this — naive classification misses real problems)

Symptom in HTML	Real cause	Text-fixable?
missing </body></html> only	LLM max_tokens cutoff mid-table	YES — append </body>\n</html>\n
no <h1> but has <h2>+	heading hierarchy weakness	YES — promote first h2→h1
no <h1> AND no <h2>+ (title-only)	semantically weak page	YES — insert <h1> from <title> after <body>
no <body> tag at all (CSS-only fragment)	LLM cutoff mid-<style> — page renders blank	NO — needs LLM regeneration
0-byte file	generation failure	NO — needs LLM regeneration

Critical mistake to avoid: classifying "fragment" by requiring <html> AND <head> AND <body> together. Files cut off mid-style have <html> and <head> opened but never reach <body> — and they have plenty of content (10-16KB of CSS). Use <body NOT in lowercase HTML as the discriminator.

Repair script pattern (anti-ai-style-factory/scripts/repair_reference_html.py)

Three classes of action:

1. Append closers — </body>\n</html>\n if missing. Cheap, browsers auto-recover.
2. Add <h1> — promote existing h2 if present, else insert from <title> after <body>.
3. Quarantine — for files that need real LLM regen, MOVE (not delete) them to styles/_quarantine/<name>/ with a _quarantine_log.json so they're recoverable. The audit then stops flagging them.

Always:

• --dry-run first to print counts and a few samples — verify classification matches ground truth before writing any files.
• Verify after by re-running --audit and confirming total_problems=0 (or close to it).
• Trigger one cron tick with hermes cron run <job_id> and read the latest output file to confirm the new state is reported.

Result on anti-ai-style-factory (2026-06-17)

AUDIT_SUMMARY audited=14623 structural=0 drift=0 ai_trace_files=12874 size_anomalies=0 total_problems=0

• Before: 14997 problems (14522 closing tags + 396 no-h1 + 472 CSS-only + 1 empty + size)
• After repair script: 0 problems
• 472 CSS-only fragments quarantined to styles/_quarantine/ for future batch resubmit
• Repair time: 2 seconds (14522 files closed, 396 h1 inserted)
• Cron tick: ok, AUDIT_SUMMARY line present in output