wiki-cron-pipeline-reconciliation

name	wiki-cron-pipeline-reconciliation
description	Audit cron batch-script runs for silently dropped inputs.
version	1.0.0
author	Hermes Agent
license	MIT
metadata	{"hermes":{"tags":["wiki","cron","pipeline","verification","data-loss","reconciliation"],"category":"wiki","related_skills":["life-screener","wiki-pipeline","wiki/inbox-screener"]}}

Wiki Cron Pipeline Output Reconciliation

Batch scoring/ingest cron scripts (score_life_inbox.py, inbox-screener variants) print a summary line like 0 入库 | 6 reject | 9 skip | 0 dup | 0 error plus per-URL log lines. When the log line count < input count, or the summary doesn't obviously match the input list, do not trust the summary — reconcile before reporting. The user demands evidence chains, not speculative conclusions; this skill is the evidence chain for "nothing was lost".

Trigger

• Summary counters don't add up against the input list (candidates.md URL count, inbox file count)
• URLs/files processed with NO log line (neither [SCORE] / [SKIP] / [FETCH-FAIL] / [QUICK-SKIP])
• You must answer "did anything get silently dropped?" after a pipeline run
• Pre-commit verification of any run that reports 0 ingests — verify the 0 is real, not a silent failure

Core Rule

No log line ≠ dropped. Batch scripts have silent branches that consume inputs without logging. The reconciliation question is: which silent branch consumed it, and is that branch legitimate? Only an input that matches NO branch AND is missing from the input file's remaining content is actually lost.

Method

1. Count inputs — candidates.md URL count / inbox file count (read the input file directly, not from memory).
2. List the logged lines — [QUICK-SKIP] / [SCORE] / [FETCH-FAIL] lines from stdout.
3. Find the silent gap — inputs with no log line.
4. Classify each silent input against the script's OWN filter constants by loading the script as a module (importlib), never by re-implementing its logic (you'll drift from its constants):

   import sys, importlib.util, urllib.parse, re
   spec = importlib.util.spec_from_file_location('sl', '/path/to/score_script.py')
   sl = importlib.util.module_from_spec(spec); spec.loader.exec_module(sl)
   bl = sl.build_blacklist()
   for u in silent_urls:
       real = sl.decode_convertkit(u) if hasattr(sl, 'decode_convertkit') else u
       nu = sl.norm_url(real) if hasattr(sl, 'norm_url') else real
       domain = urllib.parse.urlparse(real).netloc.lower()
       reason = []
       if nu in bl: reason.append('BLACKLIST')
       if any(domain.endswith(b) for b in getattr(sl, 'BLOCKLIST_DOMAINS', [])): reason.append('BLOCKLIST_DOMAIN')
       if re.search(r'utm_campaign|/lp/|/trial', real) and domain not in getattr(sl, 'HIVALUE_DOMAINS', []): reason.append('UTM/LP')
       if any(re.search(p, real) for p  (sl, , [])): reason.append()
       (.join(reason)  reason   + u)
   

Counter-Semantics Decoding (score_life_inbox.py, verified 2026-08-14)

• reject = scored-then-failed (stars≤2 / v×c<threshold / domain relevance fail) — always logged as [SCORE]
• skip = [FETCH-FAIL] + keyword-count-filtered inputs — [QUICK-SKIP] lines are logged but NOT counted
• dup = blacklist hits — silent, not logged, not counted
• Silent branches (no log): blacklist, BLOCKLIST_DOMAINS, UTM/LP heuristic, redline, life-keyword<2
• Worked math (2026-08-14): 40 URLs = 20 QUICK-SKIP(logged, uncounted) + 6 SCORE(reject) + 1 FETCH-FAIL(skip) + 5 silent-prefilter + 8 silent-keyword(skip) → 0 入库 | 6 reject | 9 skip | 0 dup | 0 error ✓

Multi-Writer Inbox Reconciliation (rss-inbox family, verified 2026-08-15)

The RSS inbox pipeline runs SEVERAL writer scripts in sequence (recovery → watchdog → curl-recovery → problem-feeds), each reporting its own "N written / M skipped". Their sums can exceed the physical file count — the writers dedup against each other at write time. Reconciliation question: is the final count a data loss, or a cross-writer collision?

Ground truth hierarchy (in order of trust):

1. Final physical count — ls raw/rss-inbox/*.md | wc -l + programmatic per-feed/per-domain breakdown (read feed_name: from frontmatter, never infer from filename).
2. curl-recovery's "N URLs already in rss-inbox/" — this is a URL-set count of the inbox at curl-recovery's start time; if it's LOWER than the naive sum of prior writers' "written" counts, the delta = cross-writer URL collisions (two writers wrote the same URL → one physical file).
3. 0-dup-URL verification — count url: frontmatter occurrences; any dup-pair count > 0 means two files share a URL (usually slug collisions from title dedup suffixes).

Known off-by-N variants:

• Off-by-1 (long documented): watchdog write was a dup of a recovery file — net contribution 0, not a missing file.
• Off-by-2 (2026-08-15, NEW): recovery-written URLs collided with problem-feeds (HF) written files — same URL, single physical file each. Signature: curl-recovery reports "N already in inbox" = naive_sum − 2, final dup-URL count = 0, all feeds still represented in the per-feed breakdown. NOT data loss — do not re-run writers to "fix" it (re-running recovery after filters re-pollutes the inbox; see rss-to-wiki-pipeline pitfalls).
• Off-by-3 (2026-08-15 07:15, combination variant): −2 cross-writer URL collisions (as above) + −1 ACTIVE-watchdog dup. When watchdog is ACTIVE (scan New>0, cascade cycle) its "1 written" is frequently a URL-dup of the same article recovery just wrote — the two writers race on the same feed entry. Net: formula 212 + 1 watchdog + 5 HF + 52 curl − 7 − 52 = 211 vs 208 physical → −3, final dup-URL count = 0. Rule: an ACTIVE watchdog adds −1 to the expected delta; a short-circuited (0 written) watchdog adds nothing. Distinguish from data loss by the same evidence chain: 0 dup URLs + per-feed presence intact + curl-recovery "already in inbox" count.
• Off-by-4 (2026-08-15 21:10, double-recovery-run variant): when a feed timeout forces a PRE-FILTER backfill re-run of recovery.py, the second run's organic captures (feed advanced in the interval) + same-URL collisions across both runs elevate the collision delta from −2 to −4. Observed: run1 194 written + run2 20 written + 5 HF + 52 curl − 7 case-study − 52 sub-1KB = 212 formula vs 208 physical → −4; run2's 量子位 +2 files were absorbed by the case-study filter (metric-number news pieces), so per-feed 量子位 held at 16 despite 20 total written. Evidence chain identical to other variants: 0 dup URLs + per-feed presence intact + feed alive verified — the elevated delta is an artifact of the double run, NOT data loss.
• Formula: final = recovery + problem_feeds + curl − case-study_rm − sub-1KB_rm. When formula ≠ physical, trust physical; explain the delta by the curl-recovery "already in inbox" count before suspecting loss.

Anticipation pattern (predicting the next run's baseline): when a DRAIN cron (wiki-inbox-scan-v2 / wechat-inbox-pipeline) ran between two scans, check its log.md entry for ingest/MERGE counts BEFORE predicting the next rebuild signature. Drain promotions of limbo articles to raw/articles drop the next recovery count by exactly the promoted set (verified 2026-08-15 twice: 04:50 drain promoted 2 → baseline 0→209 → 0→207, recovery 213→211, skipped 102→104; 09:39 run reproduced the −1 form: 07:15's Interconnects cascade article GLM-5.3 ingested at 07:37 → recovery 212→211, baseline 0→208 → 0→207, all other per-feed splits byte-identical). A recovery-count DECREASE = ingestion, not rotation or feed-health — verify by grepping raw/articles for the promoted filenames (presence confirms ingestion).

⚠️ Slug-normalization grep trap (hit 2026-08-15 09:39): when verifying ingestion by filename, the on-disk slug may differ from the article's published name because slugify normalizes special chars — glm-5.3 (dot) becomes glm-53 (hyphen) on disk. A glob like *glm-5.3* returns 0 matches → false "absent" signal that could misread an ingestion shift as rotation. Rule: grep a loose token (*glm*, *stride*, the distinctive word fragment) instead of the exact dotted title; then confirm via frontmatter (source_url + ingested date). This extends the existing shorthand-name trap to dot/hyphen normalization — exact-title globs are NEVER safe for verification.

⚠️ Organic cascade + ingestion reverse-shift co-occurrence — flat per-feed count ≠ unchanged (2026-08-20 08:22): when a drain ingestion (−1) and a scan organic cascade (+1) hit the SAME feed in the SAME cycle, the per-feed recovery count holds FLAT because the two cancel — superficially "nothing changed," actually two offsetting events. Observed: AWS-ML recovery held at 18 (ingested 1 → 17, New=1 organic → 18) while net inbox dropped −1 (0→171 → 0→170) from the ingested async-agentcore article. Disambiguate by (1) scan New>0 (organic tell), (2) watchdog 100%-hit (cascade corroborated), (3) grep raw/articles for the ingested filename, (4) git log drain commit naming the article, (5) attribute net −N to ingestion, not feed-health. A flat count is "unchanged" ONLY when scan New=0 AND no drain ran between cycles. Full evidence chain: references/2026-08-20-organic-cascade-plus-ingestion-cancel.md.

Manual-Heuristic Score Signatures

When the LLM API degrades (JSON parse failure, HTTP error, 402), scripts fall back to a keyword-count heuristic. Detect it in [SCORE] lines by the exact value×confidence products:

• v×c=30 stars=3 (1-2 life keywords)
• v×c=36 stars=3 (3-4 keywords)
• v×c=49 stars=4 (≥5 keywords)

If a [WARN] JSON 解析失败 / [WARN] DeepSeek ... line precedes one of these, the batch may be heuristic-scored — treat scores as low-confidence and check for generic-landing-page false positives (sponsor/membership pages score high on vague productivity keywords).

Pitfalls

• recovery.py per-feed fetch timeout = ONE feed silently dropped, NOT full failure (2026-08-15 21:10, rss-feed-scan): recovery.py can fail a single feed mid-run (✗ WeChat-小米技术: RSS fetch failed — The read operation timed out) while every other feed writes normally — the summary shows that feed at 0 written, N skipped (小米 18 articles vanished from the cycle). This is the same partial-output class as the fetch-problem-feeds.py end-of-run hang, but for the MAIN writer and for a MIDDLE feed (not the last). Recovery pattern: (1) verify the feed is ALIVE independently — proxy curl to its feed URL returning HTTP=200 + full size (小米: HTTP=200 size=2168034B) proves transient, not feed death; (2) re-run recovery.py BEFORE the filter chain — the "never re-run recovery.py" rule (rss-to-wiki-pipeline pitfall) applies ONLY AFTER filters run (re-running after filters re-pollutes removed files); a pre-filter re-run is the SAFE backfill and writes exactly the missing set (observed: run2 wrote 小米 18 + 量子位 2 that arrived in the interval, all other feeds 0 written, N skipped); (3) verify by per-feed physical count + mtime, not by re-reading the summary. Caveat: the backfill can surface organic content that arrived between the two runs (量子位 +2) — those get absorbed by the downstream case-study filter if they're metric-number news pieces, which is why physical per-feed count can hold flat despite more "written" this cycle (contributes to the off-by-4 variant above).
• Writer script times out mid-write = PARTIAL output, not failure (2026-08-15 07:15, fetch-problem-feeds.py): a multi-feed writer can write some feeds' files, then hang on the LAST feed (fetch-problem-feeds.py writes all HF articles first, then hangs ~300s on the Substack/SP fetch → terminal timeout kills it). The files written BEFORE the hang are valid and stay in the inbox. Recovery pattern: (1) check what's already in the inbox — ls -lt timestamps show what completed; (2) re-run the script in the BACKGROUND (terminal background=true) — its inbox-dedup skips already-written URLs, so it completes only the missing set (observed: first run wrote 4/5 HF limbo files, background re-run added exactly the 5th, voiceeq); (3) verify by filename for the known missing member, not by re-reading the summary. Do NOT re-run the whole pipeline chain — only the timed-out writer.
• Multiple timeout kills of recovery.py = STILL safe to re-run + MUST verify no corruption (2026-08-18 12:34, triple-invocation full-rebuild variant of the partial-write pitfall): recovery.py on a fresh-cycle rebuild (start inbox=0) can exceed the terminal's default 180s timeout MULTIPLE times while mid-write — this cycle it was killed at 180s (exit 124) TWICE, then completed on a 600s third run. Crucially, recovery.py is idempotent across kills: it skips any URL already in inbox/raw, so each re-run writes only the remaining set and never duplicates or corrupts. Combined output = a complete correct rebuild (observed: first killed run wrote ~195 of the expected set, final run's summary read "19 written, 276 skipped" where the 276 were already-written files). BUT a summary from the completing run is misleading (its 19 written ≠ total), so: (1) NEVER trust a single invocation's "N written" — count the physical inbox instead; (2) after the multi-run rebuild, VERIFY no partial/truncated file was left by a mid-write kill — run a frontmatter check and confirm (a killed write can theoretically leave a truncated file); (3) then proceed to the filter chain as normal, and reconcile against physical count. The tool is BLOCKED in cron mode — do this verification with via terminal, not execute_code.

References

• references/2026-08-14-life-screener-reconciliation.md — worked example: 40 newsletter candidates, 27 log lines, 13 silent URLs classified (5 pre-fetch filters + 8 keyword-filtered), summary fully reconciled, 0 data loss.
• references/2026-08-27-life-screener-38-candidates-0-ingest.md — worked example: 38 candidates → 0 ingest, 12 silent removals, ambiguous NO-BRANCH URLs (ianbarber/vocab-break, ipurple/text-template) Jina-fetched to confirm they're AI/ML/security (correctly non-life); adds the ambiguous-URL Jina-fetch confirmation loop to the method.
• references/2026-08-15-rss-inbox-multi-writer-reconciliation.md — worked example: multi-writer RSS inbox pipeline, off-by-2 between writer-sum and physical count = cross-writer URL collisions (NOT data loss), evidence chain (unique-URL count + per-feed presence + drain-log ingestion check).
• references/2026-08-15-0715-rss-inbox-offby3-and-partial-write.md — worked example: off-by-3 (collisions + ACTIVE watchdog dup) and the writer-timeout partial-write recovery pattern (fetch-problem-feeds.py 300s timeout with 4/5 files written → background re-run completes the set).
• references/2026-08-15-rss-inbox-recovery-timeout-backfill.md — worked example: recovery.py per-feed fetch timeout (小米 18 dropped) → feed-alive verification → PRE-FILTER backfill re-run → off-by-4 reconciliation; includes the 腾讯 scan New=1 count-held rotation verification via blogwatcher DB unread list.
• references/2026-08-18-rss-inbox-post-filter-phantom-and-safe-kill.md — worked example: fetch-problem-feeds SP hang killed at ~490s (HF-written-first safe-kill), plus a post-Step-5-filter HF phantom (voiceeq) that required a SECOND dedup pass to remove (count 167→166). Covers the temp-script-labeling lesson (use hermes-verify- under OS temp dir, not the repo's tracked scripts/ tree).
• references/2026-08-18-triple-invocation-recovery-rotation.md — worked example: recovery.py killed twice at the 180s default timeout then completed on a 600s third run (idempotent-across-kills, combined output clean); the required source_url corruption-verification snippet; and the rotation off-by-0 reconciliation (organic small entry replacing an outgoing big → zero collision slack, per-feed count held at 10 with a size-class composition shift).
• references/2026-08-20-organic-cascade-plus-ingestion-cancel.md — worked example: organic cascade (+1) and ingestion reverse-shift (−1) co-occurring on the same feed (AWS-ML recovery held flat at 18, net inbox −1 from async-agentcore ingestion); evidence chain to decompose a flat per-feed count before classifying.