cron-safe-git-metadata-extraction

name	cron-safe-git-metadata-extraction
description	Use when cron jobs extract new files and metadata from git.

Cron-safe Git Metadata Extraction

Trigger

• Cron report jobs: "new files committed in last N hours" + per-file metadata (score/date/vxc).
• Python scripts that shell out to git (subprocess / bash -c) for file lists or metadata.
• Any git-tracked content repo reporting: wiki entities, article catalogs, changelog generators.

Core patterns (verified 2026-08-14 on a 69-file / 10-commit window)

1. NUL-safe file list (CJK filenames)

git -c core.quotepath=false log --since="24 hours ago" --diff-filter=A --name-only -z --format= -- 'entities/*.md'

Decode in Python BEFORE splitting: raw.replace('\x00', '\n') — otherwise trailing NULs crash later subprocess calls with ValueError: embedded null byte.

-c core.quotepath=false is REQUIRED, not optional (verified 2026-08-15, 38-entity window): without it, git prints CJK filenames as octal-quoted strings ("entities/agent-\346\262\273..."), and stripping the surrounding quotes with sed does NOT recover the real name — a shell loop over those names finds 13/38 files (all ASCII-slug ones), silently dropping every CJK entity. With quotepath=false the names come out as real UTF-8 and the loop finds 38/38. Apply it to EVERY git log/ls-tree call whose output feeds filename matching.

2. Quoted pathspec with shlex.split

List-form subprocess does NOT shell-glob. Keep the quotes in the command string — git interprets entities/*.md itself. An unquoted bare glob reaches git as a literal path → silent empty output with exit 0.

3. ⚠️ %-format collision — THE trap (bit twice in one session)

Python %-formatting a command string that contains git's own --format="%s" / --format=%H raises:

• TypeError: not enough arguments for format string (from %s)
• ValueError: unsupported format character 'H' (from %H)
• ValueError: unsupported format character ' ' (0x20) (from %h %s — the space after %h is read as the format char; bit a third time 2026-08-15 on --format="%h %s")

Fix: escape as %%s / %%H / %%h %%s inside the Python format string. Audit EVERY git command containing % when building commands via %-formatting — the first fix often misses the second %-specifier in the same script.

4. Per-file ADD commit subject (authoritative score source)

git log --since="24 hours ago" --diff-filter=A --format="%%s" -- '<entity>.md' | head -1

For multi-entity commits, attribute by norm-containment + token fallback (below).

5. CJK token fallback — and its failure mode

Compressed-slug tokens (Stealing-Traces ∉ filename) recover via token containment:

toks = [t for t in re.split(r'[^0-9a-z]+', mslug.lower()) if len(t) >= 3]
if toks and all(t in n_file for t in toks): vxc = mvxc

FAILURE MODE (verified 2026-08-14): re.split(r'[^0-9a-z\u4e00-\u9fff]+', ...) does NOT split CJK phrases — Chinese has no separators, so SIGCOMM 论文复现 → ["sigcomm", "论文复现"], and the contiguous-substring test "论文复现" in n_file FAILS when the filename has 论文 and 复现 in separate positions (...复现顶会网络系统...论文自动变...). Result: silent — for a score that was actually recoverable (SIGCOMM 复现 → 72, missed by the loop, caught by manual cross-check). Mitigations: (a) also match individual CJK word tokens, (b) resolve via the metadata ledger line, (c) always spot-check top scores manually.

6. Metadata ledgers have TWO line formats — parse both

Single-line: ## [date] NEW | title | slug | vxc=NN

Multi-line:

## [date] ingest | title
- slug: entities/x + raw/articles/y
- vxc=NN (v=8 c=9 s=4)

A regex built for one format silently misses entire entries (verified: 5/5 multi-line scores missed — CDL 72, from-spec 56, prompt-to-harness 72, PhyEdit 64, file-upload 49). Extract BOTH the single-line pattern AND the - vxc= / - slug: pairs.

7. Short-token false positives from broad extraction regexes

Extracting ledger keys with ([\u4e00-\u9fffA-Za-z0-9][^\s、，,()（）]*) yields 2–3 char keys (loss, 漫谈, bi) that false-match unrelated titles:

• "Agent 评测漫谈" falsely inherited 72 (real owner: 参数化 Memory 漫谈)
• "免蒸馏，只改一个Loss" falsely inherited 81 (real owner: 重新审视交叉熵 LM Loss)

Guards: require key length ≥ 4, prefer whole-item containment, and cross-check plausibility — an entity NOT named in the 高价值 line should not get a 高价值 score.

9. Pipeline /tmp artifacts — authoritative per-entity scores when the commit subject gives only a RANGE

Batch commits sometimes carry only a score range, not per-entity values (verified 2026-08-15: 7f1ed21fe wiki-inbox-scan: 25 NEW entities + 3 MERGE ... vxc 49-81 — 25 entities, zero individual scores in the subject). The pipeline writes per-entity scores to /tmp DURING the run — check these BEFORE falling back to entity-body/log.md greps:

• /tmp/ingest-list.json — list of {"slug": <RAW article slug>, "fname": <raw fname>.md, "vxc": NN, "url": ...} for every entity the batch ingested. mtime ≈ batch commit time.
• /tmp/reuse-scores.json — rows of [channel, fname, vxc, stars] (e.g. ["rss", "10万小时训出....md", 64, 4]): scores saved by rss-feed-scan and reused by the ingest batch (its "复用 N 篇" count).

Matching pitfall — ingest-list.json keys are RAW article slugs, NOT entity filenames: raw slugs are Chinese titles (agent-harness开始自动修复系统级debug最高提升184点) while entity filenames are English slugs (agent-harness-auto-repair-debug-184) — direct norm-match on entity filename fails 22/37. Resolve each entity's sources: frontmatter to its raw slug first, then norm-match (CJK-preserving re.sub(r'[^0-9a-z\u4e00-\u9fff]','',s.lower()), containment) against the ingest-list keys. Verified: this path matched 37/37 (00:40 round) AND 27/27 same-day round 2 (commit 7f1ed21fe, ingest-list.json mtime 00:55) — index the dict by BOTH fname and slug norms so either key form hits. Both rounds' singles (glm-53 vxc=56, anthropic vxc=48) were NOT in ingest-list — those came from the log.md ingest entries; keep a small log.md override dict for single-ingest commits outside the batch.

10. Verification before delivery

• Print VXC_COVERAGE: N/M entities have vxc to see the gap before writing the report.
• Manually cross-check the top-3 scores against the authoritative ledger — automated matching alone was wrong on 4/27 scores in the verified window (Corvus 72→64, 600倍 —→56, SIGCOMM —→72, 免蒸馏 81→—).
• Run git status --short | grep '^??' every run — untracked files mean commit-step failure, not "no ingestion".

11. Publish-date field: source_published:, NOT publish_date: (verified 2026-08-15)

Cron prompts often say grep "publish_date:" raw/articles/... — the actual field in wiki raw articles is source_published: in the frontmatter (grep -m1 "^source_published:"). publish_date:/publish_time: return EMPTY on every file. Fallbacks:

• Some articles lack source_published entirely (e.g. merged/manual ingests) — fall back to ingested: and note it's the ingest date, or mark the date as —.
• Don't trust the prompt's field name; inspect one article's frontmatter (head -20 raw/articles/<slug>.md) once per pipeline to confirm the real field.

12. Resolve entity → raw article via sources: frontmatter (authoritative)

Entity filenames and raw-article slugs diverge (English slug entity vs Chinese-title raw slug, or truncated titles). The one-line sources: frontmatter in the entity file IS the resolver:

src=$(grep -m1 "^sources:" "$f" | sed 's/^sources:[[:space:]]*\[//;s/\].*//' | tr ',' '\n' | head -1 | sed 's/raw\/articles\///;s/[[:space:]]//g')

Take the FIRST entry when multiple (sources: [a, b] — 2nd source merges); strip raw/articles/ prefix and whitespace. This is more reliable than norm-matching entity filename → raw slug (which the 2026-08-15 window showed fails when titles are truncated in the entity slug).

Pitfalls

• Use a UNIQUE, session-scoped /tmp script name (e.g. /tmp/extract-<job>-<date>.py), never a generic one like /tmp/extract-new-entities.py — sibling subagents / parallel cron jobs writing the same generic name get their file silently overwritten (observed 2026-08-15: write_file warned "modified by sibling subagent"). A unique name avoids the collision entirely and costs nothing.
• read_file may misdetect CJK-heavy ledger files as binary — use grep/sed or Python with encoding='utf-8', errors='replace'.
• In cron mode, never pipe extraction-script output into a second interpreter (python3 x.py | python3 -c "..." is blocked by the security scan); print the final formatted table inside the script.
• Inline python3 -c "<multi-line script with subprocess/git>" is ALSO guard-blocked (verified 2026-08-15): a python3 -c containing a subprocess.run(...) git call was rejected with Blocked: command or referenced script cannot restart or stop the gateway... — a false positive on the quoted command content, not a real gateway operation. Fix: write_file the script to /tmp/<unique-name>.py and run python3 /tmp/<unique-name>.py. When a one-liner is refused, go straight to the file form — don't try variations of the inline form.
• head -20 on the raw --name-only list truncates INSIDE a multi-file commit — re-run with -z and no head for counting; only cap the final report table. For the FULL per-commit file list (not just a count), use git diff-tree — git log <sha> walks the whole ancestry, but diff-tree is single-commit truth:

  git diff-tree -r --no-commit-id --name-only -z --diff-filter=A <COMMIT> -- 'entities/*.md' | tr '\0' '\n' | sort
  

  CJK-safe (no core.quotepath needed), no head. Verified 2026-08-15: commit 7f1ed21fe claimed 25 NEW + 3 MERGE but head -20 showed only 15; diff-tree recovered all 25.

Supplied tools

• references/verified-window-2026-08-14.md — the 69-file/10-commit transcript: ledger line formats (高价值 / domain-reject 入档 / HF batch **10 NEW**), exact regexes that missed or false-matched, and the corrected attribution table.
• references/verified-window-2026-08-15.md — the 38-entity run: quotepath=false 13/38→38/38 fix, full working extraction chain (sources frontmatter → source_published → 3-tier vxc lookup), inline python3 -c guard false-positive transcript.
• references/verified-window-2026-08-15-round2.md — same-day round 2 (27 entities, 27/27 via ingest-list.json): both-key norm dict (fname+slug), find_raw_file exact-then-containment, singles-override dict for log.md-only scores, ingest-list.json-is-a-list gotcha.