wiki-status-report

name	wiki-status-report
description	Generate periodic status reports for the Obsidian wiki — gather directory stats, discover recently committed entities, extract publish metadata from raw articles, and format a deliverable report. Covers daily/weekly cron-driven automation and ad-hoc health checks.
category	wiki

Wiki Status Report

Generate a periodic health report for the wiki: entity/article counts, inbox backlogs, recently ingested entities with their publication metadata and vxc scores.

Trigger

• Cron job: wiki-inbox-scan-v2 runs this on a schedule (every 20–120 min).
• Ad-hoc: User says "生成 Wiki 状态报告" or similar.

Output

• Normal: A Markdown report with stats table + new-entities table.
• Silent: If zero new entities committed in the last 24 h, output [SILENT] alone to suppress delivery.

Step-by-step (cron / manual)

0. Resolving entity → raw article path

The entity sources: frontmatter field links to raw articles. Two formats:

Inline array (most common):

sources: [raw/articles/some-slug]

Block list (used by synthesised/deepen entities):

sources:
  - raw/articles/slug-one
  - raw/articles/slug-two

Always resolve sources: first rather than assuming the entity filename matches the article filename — they often diverge (e.g. entities/market-surveillance-agent-langgraph-strands-agentcore.md → raw/articles/market-surveillance-agent-with-langgraph-and-strands-on-agen.md). Use the scripts/resolve-raw-article.sh script or scripts/report-new-entities.sh which handles all formats.

Pitfall — do NOT extract the source slug with grep -oE 'raw/articles/[a-zA-Z0-9-]+': the ASCII character class silently truncates at the first CJK character (raw/articles/agent instead of raw/articles/agent开始自我进化会出题会反思还会自己长出新技能), producing a wrong path that fails [ -f ] and grep checks with no error. The canonical extraction that keeps Chinese slugs is the sed-strip form (it matches the whole line containing raw/articles then strips brackets/prefix, preserving CJK):

src=$(grep -m1 -A1 "^sources:" "$ef" 2>/dev/null | grep -E "raw/articles" | head -1 | sed 's/^ *- *//; s/^sources: *\[*//; s/\]$//')

For inline-array frontmatter (sources: [raw/articles/中文-slug]), grep -A3 "^sources:" and read the raw line directly — the bracket form is a single line, no list parsing needed.

Pitfall — some sources: entries INCLUDE the .md extension (observed 2026-08-13, RSS-recovery batch: sources: [raw/articles/001参数就能接近全量微调低数据微调极致省参方案来了-acl26.md]). If your resolver appends + ".md" to a slug that already carries it, you get slug.md.md → [ -f ] false-negative on files that exist, and every metadata grep silently returns nothing. Always strip a trailing .md after extraction: Python slug.removesuffix(".md"), bash ${src%.md}. The canonical sed-strip form above does NOT do this, so batch loops that concatenate .md must handle both variants.

Pitfall — sources: values are FULL PATHS (raw/articles/slug), so Python resolvers that blindly os.path.join(WIKI, 'raw', 'articles', slug + '.md') build raw/articles/raw/articles/slug.md (verified 2026-08-13): the extraction succeeded (title + sources parsed) but every raw lookup silently missed, producing an all-— vxc/date table. Normalize the extracted value to a bare slug BEFORE joining — slug = src.replace('raw/articles/', '') then slug.removesuffix('.md') — or branch: if src already starts with raw/articles/, join only os.path.join(WIKI, src + '.md'). The first metadata pass failing 100% across 44 entities is the signature of this bug: titles render, vxc/date are all —, and the raw files verifiably exist on disk (spot-check with grep -m1 '^source_published:' "raw/articles/<slug>.md" before debugging the loop).

Some entities have multiple source articles (e.g. regression-tax-skills-hurt-llm-agents.md references two articles). In that case, the first source in the list is the primary one — use it for metadata extraction.

0. Touch heartbeat

python3 ~/wiki/scripts/cron-heartbeat.py touch wiki-inbox-scan-v2

1. Collect directory stats

Use find (not ls with glob) to avoid "argument list too long" on directories with 3000+ files.

# entity count
find ~/wiki/entities -maxdepth 1 -name '*.md' 2>/dev/null | wc -l

# article count
find ~/wiki/raw/articles -name '*.md' 2>/dev/null | wc -l

# inbox backlogs
ls ~/wiki/raw/rss-inbox/ 2>/dev/null | wc -l
ls ~/wiki/raw/wechat-inbox/ 2>/dev/null | wc -l
wc -l ~/wiki/raw/email-inbox/candidates.md 2>/dev/null

2. Discover new entities

Use git -c core.quotepath=false to avoid quoting issues with Chinese/CJK filenames. Combine --name-only with the commit log so one command returns both messages and filenames. Cap output with head -20 for safety on busy periods.

cd ~/wiki
git -c core.quotepath=false log --since="24 hours ago" --oneline --diff-filter=A --name-only -- 'entities/*.md' | head -20

Pitfall — Do NOT pipe through grep '^entities/': Chinese filenames appear as quoted lines and will be silently dropped. The git -c core.quotepath=false flag prevents quoting entirely. If you must filter for entity paths only, use grep -E '(^|"?)entities/' instead. Count-mismatch tell — when grep '^entities/' | sort -u returns FEWER lines than the raw --name-only stream, CJK entities were dropped; the delta IS the drop count (verified 2026-08-22, 13-entity window): plain git log --since="24 hours ago" --diff-filter=A --name-only --pretty=format: -- 'entities/*.md' | grep '^entities/' | sort -u yielded 11, while the unfiltered --name-only stream carried 13 — the 2 missing were the Chinese-named entities (agent-终章harness-成本篇一次百炼账单降低-88-实战.md, 端到端论文生成系统假结论检出率92自动跑实验画图直出论文初稿.md). The cron prompt's canonical | grep entities/ | head -20 is the same trap. Fix: write the unfiltered --name-only stream to a file, then process in a write_file'd Python script that unquotes octal escapes (s.encode().decode('unicode_escape').encode('latin1').decode('utf-8')) and dedupes in Python — this recovered 13/13. Trust the raw stream's line count, not the grepped one.

Pitfall — the git-log file list ALSO leaks log.md DIFF content lines that merely mention entities/ text (verified 2026-08-22): when log.md is among the commits in the window, git log --diff-filter=A --name-only -- 'entities/*.md' | grep 'entities/' catches log.md's - NEW entities/<slug> (vxc=NN) summary bullets as if they were entity paths — a false "file" per logged ingestion. The grep -E '(^|"?)entities/' filter excludes them (those lines start with - , not entities/ or a quote), so prefer it. If you DO see such lines in an unfiltered grep, they're not garbage to discard — each is a (vxc=NN)-bearing log.md bullet for an entity in the CURRENT window, i.e. a free vxc source for entities whose commit subject/raw frontmatter carry no score (observed this window: agent-终章harness vxc=64, measuring-benchmark vxc=56, 端到端论文 vxc=49 recovered this way).

Pitfall — git log -z GLUES the commit header+message onto the first filename chunk (verified 2026-08-13): output is NUL-separated, but the first filename of each commit is concatenated to the message text (...3 deterministic skip \n\nentities/how-...md\0). Splitting on \0 and filtering startswith("entities/") then SILENTLY drops that first file per commit (8 new entities → only 3 survived a naive split). Fix: pass an EMPTY --format= so -z yields a pure NUL-separated filename stream:

git log --since="24 hours ago" --diff-filter=A --name-only -z --format= -- 'entities/*.md' | tr '\0' '\n' | grep '^entities/'

-z mode also avoids the octal-quoting of CJK filenames that plain --name-only emits — no core.quotepath needed.

Empty result ≠ no ingestion — check for untracked files before concluding [SILENT]. A --diff-filter=A query returning empty does NOT mean the last 24h had no ingestion. Observed 2026-08-03: log.md recorded 08-02 ingestion (OpenWorker NEW vxc=48, YC-QM RAW vxc=30, Milvus NEW vxc=48, Metis RAW vxc=30, MCP v2 RAW vxc=35, ICML RAW vxc=35) but git log --diff-filter=A was empty because the pipeline's commit step silently failed — every entity/raw file was ?? untracked in the worktree. Diagnose with:

git status --short | grep '^??'         # untracked files = real but uncommitted work
git ls-files | grep -i <slug>           # empty = file never committed
git log --all --oneline -- '<path>'     # empty = never in git history at all

If log.md has dated ingest entries AND files exist on disk but git ls-files is empty: report the ingestion (log.md is authoritative) with a prominent ⚠️ anomaly flag — "files never committed, worktree has N untracked + M modified files". Do NOT output [SILENT]; the user needs to know the commit step failed. Also scan git status --short modified files — an in-progress batch (e.g. citation ^[raw/...] fixes touching dozens of entities) explains a dirty worktree and is worth one line in the report.

• The commit-step failure REPEATS across consecutive batches — re-scan untracked files every run, even when --diff-filter=A has hits. Observed 2026-08-05: the 08-04 batch (STAROps NEW vxc=56 + 6 RAW/SUPP articles + codexclaude-code entity) was logged in log.md but never committed — identical to the 08-02 batch flagged on 08-03. One committed entity (MirrorCode) did NOT mean the batch was clean: 19 wiki files sat untracked next to it. When this failure mode appears once, expect it to recur; git status --short | grep '^??' is mandatory every run.
• Stale batches ACCUMULATE — the untracked scan is a running backlog of every batch since the last successful commit. Third confirmation 2026-08-05 (evening run): BOTH the 08-02 batch (OpenWorker vxc=48, Milvus vxc=48, YC-QM vxc=30, Metis vxc=30, MCP v2 vxc=35, ICML vxc=35) AND the 08-04 batch (STAROps vxc=56, FDX vxc=49 SUPP, 5× RAW vxc=30-35) sat untracked simultaneously, plus the codexclaude index-entry variant. Count the FULL untracked wiki-content set (4 entities + 15 raws = 19 files) and report every batch's scores — one committed entity in the window does not clear the ledger for older batches.
• Fourth confirmation 2026-08-06: the backlog grew while NEW entities still committed fine. 31 untracked = 29 wiki content (5 entities + 24 raws, now spanning the 08-02/08-04/08-05 batches: OpenWorker, Milvus, STAROps, WorkBuddy, codexclaude-code + 24 raws) + 2 non-wiki (scripts/kg-quiz-eval/, scripts/step0-clean.py). Same run had 17 new entities committed cleanly — so a working commit step does NOT clear the untracked ledger; the failed commit(s) happened at an earlier point in the pipeline and nothing re-commits them. Keep reporting the anomaly every run; the split-quote pattern (grep -cE 'entities/|raw/articles/' for the wiki subset, then a second grep for the non-wiki remainder) is the fast way to characterize 30+ untracked files.
• New anomaly variant — index entry committed, entity/raw files never committed: commit 29d9cd13f added the index.md entry for codexclaude-code (as a MISSING-index fix) while entities/codexclaude-*.md and its raw article stayed untracked forever. An index.md hit does NOT imply the entity is in git — verify with git ls-files | grep <slug> (empty = never committed). Also note the recovery path worked: entity had rating: v7c8 frontmatter and the raw article had source_published: 2026-07-28, so metadata was fully recoverable despite the uncommitted state.
• Categorize untracked files before quoting a count: the raw git status total mixes wiki content with non-wiki files (, , snapshots). Observed 26 untracked = 4 entities + 15 raw articles (the real anomaly, reportable) + 7 non-wiki (minor, one line). Quote the wiki-content subset and its entity/raw split, not the raw total — an inflated number buries the actual risk.

3. Tooling constraint (cron mode)

execute_code and patch are blocked when running as a cron job. You cannot use Python via execute_code to loop over entities or batch-extract metadata, nor use patch for file edits. All edits to skill files must go through skill_manage(action='patch').

Efficient batch path that IS cron-safe: write_file a bash script to /tmp, then bash /tmp/script.sh via terminal. Verified 2026-08-11: this replaces 5-10 manual read_file/grep round-trips with one script run, and dodges the terminal-guard quirk (inline bash -c loops containing CJK get hard-blocked). Keep the script self-contained (cd ~/wiki at top) and print ===== label ===== separators per entity so output stays parseable.

Pitfall — a multi-file bash for loop listing several long CJK raw/articles/... paths DIRECTLY as the terminal command (no bash -c, no script) ALSO trips the cron hardline blocklist (verified 2026-08-19): the command for a in raw/articles/中文-A.md raw/articles/中文-B.md ...; do grep "$a"...; done was rejected with BLOCKED (hardline): command parser limit or malformed executable payload ... saved to ~/.hermes/cache/blocked-scripts/blocked-*.sh, exit -1. This is a DIFFERENT trigger surface from the bash -c CJK block — it fires on oversized/unparseable one-liner payloads, and it fires per-command: retrying the same inline shape fails again. Two cron-safe recoveries: (1) write the loop to /tmp/x.sh and run bash /tmp/x.sh (the documented path); (2) simplest for a handful of files — issue ONE simple grep -m1 -E '^source_published:' '<path>' per file as separate parallel terminal calls. Single-file greps with CJK paths pass the guard fine; only the compound multi-path loop trips it. Don't read the hardline block as "grep is blocked" — escalate from compound loop → single-file grep (or script), not from grep → nothing.

Python variant equally cron-safe, preferred for CJK-heavy extraction (verified 2026-08-13): write_file /tmp/extract.py + python3 /tmp/extract.py works identically and handles Chinese filenames/json.dumps(ensure_ascii=False) far more robustly than bash string surgery. Use a job-unique filename (e.g. /tmp/wiki-status-extract-0827.py) — sibling cron agents share /tmp and overwrite a plain /tmp/extract.py (observed 2026-08-15: write_file returned 'modified by sibling subagent' warning on a bare /tmp/wiki-status-extract.py). Pitfall — the <jobid> token used throughout this skill's command examples is a PLACEHOLDER you must SUBSTITUTE with a real unique string (date / PID / timestamp), never copy verbatim into an actual filename (hit 2026-08-27): write_file to /tmp/wiki-status-extract-<jobid>.py creates a file whose path literally contains </>, and then python3 /tmp/wiki-status-extract-<jobid>.py FAILS in bash because angle brackets are redirection operators (jobid: No such file or directory, exit 1, before the script even runs). Recovery: copy to a clean name with quoting — cp '/tmp/wiki-status-extract-<jobid>.py' /tmp/wiki-status-extract-0827a.py && python3 /tmp/wiki-status-extract-0827a.py. Iterate: write script → run → inspect JSON → patch script (not shell one-liners). Beware execute_code is still blocked in cron mode — use write_file + terminal python3, never execute_code. Pitfall — a NON-GREEDY +? char class in a hand-written Python resolve_sources silently captures ONE CHARACTER per slug (verified 2026-08-22): re.search(r'raw/articles/([^\s\]\]",]+?)(?:\.md)?', line) returns raw=c, raw=v, raw=h — one leading letter per path — so every raw lookup misses and the whole vxc/date pass collapses. Use a GREEDY class ([^\s\]\]",]+) and strip .md/trailing punctuation after, or reuse the canonical sed-strip form from step 0. Tell: the extraction prints a per-entity raw=<single-char> column. Recovery: vxc from log.md (authoritative v×c=NN ingest entries) + dates via direct single-file grep -m1 -E '^(source_published|publish_date|published|created):' raw/articles/<slug>.md — don't debug the loop.\n ( on e.g. ) even though the identical pattern compiles in a standalone probe (verified 2026-08-20). The file shows can be intact while executes different/corrupt bytes — so do NOT burn round-trips debugging the regex. Two clean recoveries: (1) re-write the script under a genuinely unique suffix ( — a static label like is NOT enough; sibling wiki-status agents can collide on it), or (2) pivot to the single-file bash path, which is documented cron-safe and sidesteps the python interpreter entirely. : it covers the NUL-decode crash, the glob trap, the -expansion trap ( → silent empty output; expand to the absolute path inside the helper — a naive doubles it to ), the commit-score paren form, the compressed-slug token fallback, §12 regex backslash over-escape in write_file'd scripts (silent all- while titles render; grep the file for before running), and §13 the one-occurrence bare fallback for parenthesized single-entity ingests — all hit across the 59/28/8-entity windows.

Pitfall — python3 /tmp/x.py | python3 -c "..." is BLOCKED in cron mode (verified 2026-08-14): piping extraction-script stdout into a second interpreter triggers the tirith security scan (HIGH: pipe to interpreter) and the command hangs in pending_approval — it never runs. Fix: print the final formatted table INSIDE the extraction script (append a table-printing section, no json.dumps-only output) and run python3 /tmp/x.py bare. Never pipe script output into another interpreter in cron mode; inspect-and-format in the same script instead.

Pitfall — read_file misdetects log.md as a binary file (verified 2026-08-13, 174 KB UTF-8 with mixed CJK): read_file ~/wiki/log.md returns Binary file - cannot display as text and yields zero lines, stalling log.md-led analysis. It is NOT actually binary — grep/sed read it fine, and the parse is clean. Cron-safe workaround: write_file /tmp/extract-log.py that opens with encoding='utf-8', errors='replace' and prints the section you need, or plain grep -n '^## \[2026-08-13\]' log.md / sed -n '696,724p' log.md when line numbers are known. Do not conclude log.md is corrupt — it's a read_file detection quirk.

4. Extract metadata per entity

For each new entity, collect:

• Title: from entity frontmatter (grep "^title:" entities/<file>)
• Entity file on disk: always verify [ -f "$entity_file" ] — git log --diff-filter=A can report false positives from merge commits or octal-encoded mismatches.
• Raw article existence: check TWO things:
  1. On disk: [ -f "raw/articles/$slug.md" ] — may exist if the pipeline cached it
  2. In git: git ls-files "raw/articles/$slug.md" — may be empty even if disk exists (cached but never committed). If raw article is on disk but not in git, its metadata (vxc, source_published) may be absent.
• vxc (priority order by entity origin): Ingest-pipeline entities (committed by cron: wiki-inbox-scan or similar):
  1. Commit message — most reliable for pipeline-ingested entities: git log --oneline -- 'entities/<file>' | head -1 and look for vxc=NN or heuristic vxc=NN. Prefer the ADD commit, and never blind-grab the first vxc= in the subject (verified 2026-08-13, 8/8 entities cross-checked): multi-ingest commit subjects list several pairs — 00077a3a0 = (tiered-kv-cache NEW vxc=64, oneadvanced-uk-sovereign NEW vxc=56, solv-labs MERGE vxc=64). A first-match regex grabbed tiered's 64 for oneadvanced (true: 56) and WorldTrace's 64 for rd-signal-2 (true: 49). Slug-overlap matching also fails — messages use COMPRESSED slugs (oneadvanced-uk-sovereign) that are not a contiguous substring of the filename (how-oneadvanced-deployed-over-50-ai-agents-on-uk-sovereign-aws). Authoritative per-file query returning ONLY the ingest commit's subject:

  git log --since="24 hours ago" --diff-filter=A --format="%h %s" -- "entities/<file>.md"
  

  When it lists multiple pairs, attribute by normalized-slug overlap (re.sub(r'[^a-z0-9]','',name.lower())); if the message slug is compressed/ambiguous, report the value the ADD commit names for that entity and cross-check log.md (v×c=NN). Full detail: references/multi-ingest-commit-vxc-association.md. Pitfall — the first-match trap ALSO fires via a Python git log -z commit-map, not just the bash/grep path (verified 2026-08-22): I built from , then ran over the JOINED per-file message. On the shared commit , BOTH files' joined messages contain both pairs, so first-match grabbed 56 for BEACON too (true: 72). The Python map made the two-entity attribution look per-file but wasn't — it inherited the same misattribution. Fix is identical to the bash path: attribute within the commit by normalized-slug overlap (), or do the authoritative per-file query and read that entity's own pair. Any score you can't attribute by slug must be cross-checked against before it enters the table. (verified 2026-08-15, recovered 25/25 on a 27-entity window): the wiki-inbox-scan run writes a JSON list of keyed by RAW filename ( → 56, → 72). Match the entity's resolved raw slug by CJK-preserving norm containment BOTH ways (). Without this file, 23/27 entities render . Also: single-entity ingest subjects in PARENTHESIZED form () FAIL the pair regex — the before breaks token-whitespace-vxc, so a pairs-only parser reports for a scored entity. A bare fallback is safe ONLY when has exactly ONE occurrence (multi-pair subjects carry several and must stay slug-matched to avoid the first-match misattribution trap). (observed 2026-08-13): on Chinese titles it strips every character — → (empty key) and → . In a log.md title-matching loop, two unrelated all-CJK entities (, ) both normalized to and inherited 相关性≠因果性's vxc=72 via the empty-key collision — false scores that survived into the report table. For CJK title matching use a CJK-preserving norm: — and prefer containment () over exact-key dict lookup. See . 2. — field (set by scoring pipeline, but often absent). (verified 2026-08-22): arxiv-report and getDX-type raws carry (plus // sub-fields) instead of / — TaoLive arXiv 2608.15763 and getDX Q2-2026 report both resolved to . Add to the frontmatter grep set. Note a -carrying raw does NOT imply the whole batch is scored — in the same 10-entity window the other 8 (WeChat/RSS) had no score at all (see the uniform-absence → N/A rule below).\n (verified 2026-08-27, 技术号 batch: gaode-supply-quality 8/9/72, gaode-push 8/9/72, workspace-baidu 8/8/64, tencent-four-gear 8/9/72). Grepping only or misses them → all- table. Extract with and use the value (a bare number like /) for the table; the / pair is the two-axis breakdown. These same curated raws carry + and NO (see the curated-raw date rule) — so a scored-but-dateless batch of 技术号 raws is the signature of this field layout. 3. — in body text (Unicode ×, e.g. or ) (committed by or batch auto-expand):

4. Format report

Use the template at templates/daily-status-report.md. It expects these fields:

• {{DATE}}, {{ENTITY_COUNT}}, {{ARTICLE_COUNT}}
• Inbox backlog counts
• A table of new entities with title, vxc, and publish date

Grouping convention (based on commit type):

• ingest commits (vxc in message): show vxc scores directly
• deepen commits (batch auto-expand): may or may not have vxc — check entity body (v×c score:) and raw article (score_vc:) before concluding —. Some deepen entities are fully scored with raw articles and source_published; others are pure stubs. Split by sources: presence, not by commit message.
• 360° deep-optimization / batch-fix commits (e.g. "Wiki 360° deep optimization: 0 errors, type/naming/sources/MOC comprehensive fix"): commit 5+ entities at once with NO vxc in the commit message. These are typically WeChat-derived entities whose scores live only in log.md (v×c=NN) or as rating: vNcM frontmatter, and whose publish date must fall back to entity created: with ~ prefix (no source_published). Check log.md + rating: before reporting —.

Massive batch windows (100+ new entities in one commit)

A single "Wiki 360° optimization phase N" commit can add 300+ entities at once (observed 2026-08-01: a93c4815f added 318 of a 327-entity window). The canonical git log | grep entities/ | head -20 then returns 20 rows ALL from the same batch with — vxc — a useless, misleading headline. Handle like this:

1. Count added entities per commit first to see if one batch dominates:

   git -c core.quotepath=false log --since="24 hours ago" --name-status --format="COMMIT %h" -- 'entities/*.md' \
     | awk '/^COMMIT/{c=$2; count[c]=0} /^A\t/{count[c]++} END{for (c in count) print c, count[c]}'
   

   Pitfall — git log <commit> --diff-filter=A walks the ENTIRE ancestry, not one commit (verified 2026-08-13): git log 04479d0e0 --diff-filter=A --name-only returned 49 entity paths spanning 5 different commits (tiered-kv-cache, rd-signal-2, worldtrace, tmall, cuda leaked in from sibling commits) when that commit actually added exactly 39. git log <sha> always includes every ancestor's changes; there is no way to make it per-commit. Single-commit truth is git diff-tree:

   git diff-tree -r --no-commit-id --name-status --diff-filter=A <commit> -- 'entities/*.md'   # count + status
   git diff-tree -r --no-commit-id --name-only -z --diff-filter=A <commit> -- 'entities/*.md' # CJK-safe NUL stream
   

   The -z variant pairs with the same tr '\0' '\n' decode as the git log -z pattern (section 2) and needs no core.quotepath. Use it when a batch window needs per-commit file lists: loop commits from the awk counter, diff-tree each one, and the totals must sum to the awk count — if they don't, you've mis-attributed files to the wrong commit.
2. Prioritize the real cron-ingest entities (the handful with vxc in commit messages — NVIDIA/GenRec/etc.) as the featured table; treat the batch as a separate "批量优化新增" section with a ~8-10 row representative sample, not 20 indistinguishable rows.
3. Batch entities can still carry vxc — from raw article frontmatter or log.md matches (46/327 in the observed window). The script's stderr line VXC_COVERAGE: N/M entities have vxc tells you the coverage; awk -F'\t' '$3 != "—"' /tmp/new-entities.tsv | sort -t$'\t' -k1 -rn | head gives the true top scores across the whole window (vxc column is a bare number, no vxc= prefix — don't grep 'vxc=NN' on the TSV).
4. Publish dates on batch entities: many resolve to 2026-07-01-ish from source_published (raw articles predate the entities — entity filename has 2026-05-XX prefix but that's the article date, not the commit date). Report the raw article's date; leave — when absent.
5. Recovering top batch scores WITHOUT the TSV script (direct loop, when you don't want to hand off to ): resolve → raw article → / frontmatter and sort. Verified working 2026-08-01 on the 318-entity phase-2 window — a meaningful share of batch entities carry in their raw article frontmatter, so top-10 scores are recoverable even when entity bodies are bare: Pitfalls hit while building this:

5. Silent-or-deliver

if len(new_entities) == 0:
    print("[SILENT]")  # suppress delivery

The system automatically delivers the final response to the configured destination (email, etc.) — do NOT call send_message or a mailer.

Edge cases & pitfalls

• Chinese filenames: git log --name-only emits octal-escaped paths. Use git -c core.quotepath=false log ... or read the file directly from disk.
• Entity exists on disk but git path is octal: The file is findable by glob. Just head the file direct by glob-resolved path.
• Deepen batch has no vxc in commit message: Look for v×c= (Unicode ×) in entity body text, v×c score: in entity body blockquote, score_vc: in raw article frontmatter, or review_value / review_stars in entity frontmatter. Deepen entities CAN have full scores — don't default to —. Check Patterns H and I in the reference file.
• Raw article may not exist on disk: The inbox pipeline sometimes deletes the raw article after entity creation. Fall back to reading from git history, or report — for metadata.
• vxc in commit message is the most reliable signal for ingest-pipeline entities. For deepen/expand entities, look in entity body instead.
• rating: field ≠ vxc: Entities with rating: v9c9 or rating: v8c7 in frontmatter use a two-axis scoring system (value×confidence) that is NOT the same as vxc. Do NOT compute 9×9=81 and report it as vxc. Report as rating=v9c9 instead. See references/entity-vxc-discovery-patterns.md Pattern F.
• macOS grep has no -P: grep -P (PCRE) is GNU-only and dies on macOS with grep: invalid option -- P. Use grep -E / grep -oE for extended-regex extraction, or sed -n for line slicing. Never write extraction commands with -P — they will fail the whole loop when the first entity hits them.
• macOS bash is 3.2 — declare -A (associative arrays) FAIL with declare: -A: invalid option, and the failure is SILENT: the script keeps running but every lookup returns the LAST-assigned element, so all keys resolve to one wrong path and every result looks plausible. Observed 2026-08-11: 5 raw-article paths all resolved to the final key's path, so every entity reported "IN HEAD: yes" from the wrong file. Use paired strings in a plain array — arr=("key|path" "k2|p2") — and split with / .

Supplied tools

File	Purpose
scripts/report-new-entities.sh	Bash pipeline: TSV output of new entities with metadata
scripts/report-new-entities.py	Python equivalent (Unicode-safe)
templates/daily-status-report.md	Canonical report template with placeholders
references/entity-vxc-discovery-patterns.md	All vxc locations across entity types (Patterns A–I)
references/entity-article-filename-divergence.md	Entity filename ≠ article filename patterns and resolution strategies
references/vxc-source-published-extraction.md	Extraction commands for vxc + publish date
references/multi-ingest-commit-vxc-association.md	Correct vxc attribution when ONE commit ingests 2–3 entities (per-file --diff-filter=A query, compressed-slug trap, git log -z filename gluing) — verified 2026-08-13
references/python-extraction-script-patterns.md	Python-port pitfalls for the cron-safe extraction script (verified 2026-08-14/08-15): git log -z NUL-decode crash, shlex.split glob trap + ~-expansion trap (git -C ~/wiki silently empty; fix cmd.replace("~/wiki", os.path.expanduser("~/wiki"))), NEW(n)/RAW(n) commit-score paren form, compressed-slug TOKEN fallback, head -20 mid-batch truncation
references/raw-metadata-recovery-from-git-head.md	Verified cron-safe batch-loop pattern (write_file /tmp + bash), declare -A silent-failure trap, [ -f ] false-negative ladder, git-HEAD recovery of deleted raws, git-history recovery when raw absent from disk AND HEAD (Cause 3, git log --all --format=%H -1 + git show), newsletter body-byline date fallback (2026-08-11/08-12)
references/review-value-stars-pattern.md	review_value / review_stars frontmatter for deepen entities
references/cjk-title-matching-vxc-attribution.md