wiki-entity-metadata-extraction

name	wiki-entity-metadata-extraction
description	Use when extracting wiki entity vxc scores and dates.
category	wiki

Wiki Entity Metadata Extraction

Extract per-entity metadata — title, vxc score, publish date, rating — for newly-committed wiki entities, reliably across CJK filenames, multi-source entities, multi-entity commits, and batch windows. This is the extraction engine behind status reports (wiki-inbox-scan-v2 cron), audits, and quality passes.

When to use

• Any task needing a per-entity table of title | vxc | publish_date for entities added in a time window.
• Cron status reports, wiki audits, batch-verification passes, quality-improvement sweeps.

The pipeline

1. CJK-safe new-file list (never plain git log | grep entities/ — octal-quoted CJK lines get dropped):

   git log --since="24 hours ago" --diff-filter=A --name-only -z --format= -- 'entities/*.md' | tr '\0' '\n' | grep '^entities/'
   

   -z --format= yields a pure NUL-separated filename stream; no core.quotepath needed.

2. Read log.md WHOLE, never grep-filtered by date pattern. Score facts live in continuation lines that contain no date: the - 高价值：... line and - N 篇 domain-reject 入档：... line sit under a dated ## header but themselves have no date. grep "2026-08-1[34]" log.md returns 29 lines and silently drops both score-bearing lines → 0 score facts. Open the file with Python open(..., encoding='utf-8', errors='replace') and iterate every line. (read_file misdetects log.md as binary — same workaround family.)

3. Per entity, parse frontmatter: title, created, rating, and sources: — which may be inline array (sources: [raw/articles/slug]), block list (sources:\n - raw/articles/a\n - raw/articles/b), carry .md suffix, or full raw/articles/ prefix. Normalize each to a bare slug with a two-pass resolver: first try same-line inline-array regex ^sources:\s*(.+)$, then a dedicated block-list regex ^sources:\s*\n\s*-\s*(raw/articles/[^\s\]]+) — then strip, in order: sources: prefix, - list marker, leading [, raw/articles/ prefix, trailing ], trailing .md. (All four traps below are verified: each stripping step is mandatory, in that order.)

4. vxc priority chain (stop at first hit):

   1. Commit subject — most reliable. Split into two cases:
      • Single-entity commit (commit adds ≤1 entity file): any vxc=NN, or (vxc=NN) in parens, or NEW(NN)/RAW(NN)/MERGE(NN) without vxc= — all are that entity's score. Regex: v[×x]c=(\d+) or (?:NEW|RAW|MERGE|SUPP)[（(](\d+).
      • Multi-entity commit (subject lists several pairs, e.g. (prompt-to-harness vxc=72 / handroid vxc=56 / file-upload-multimodal vxc=49)): match each name (vxc=|NEW[(]) pair against the entity basename norm — re.sub(r'[^0-9a-z\u4e00-\u9fff]','',name.lower()) containment. Never blind-grab the first vxc= in the subject.
   2. Batch-run /tmp JSON artifacts — when a cron-ingest batch commit subject carries only a vxc RANGE (vxc 49-81) and log.md only summarizes (11 passing (vxc 49-81)), the per-article scores live in the batch run's own /tmp files. Try in order: ingest-list.json (the ingest ledger: list of dicts {slug, fname, vxc, url} — key by fname; matched 22/27 entities directly on the 2026-08-15 round), then score_results_MMDD.json (API LLM output: [{fname, title, value, confidence, stars, vxc, reason}]) and reuse-scores.json (reused rss-feed-scan saved scores: [[source, fname, vxc, stars], ...]). If both jsons miss an entity, the batch log.md 新入档 URL（评分一次…） block lists per-URL (vxc=NN INGEST) lines — last-resort recovery (recovered voiceeq 56 when ingest-list/reuse both lacked it). Resolve each entity's sources: raw slug and key by exact fname — no normalization needed. Verified 39/39 coverage on a 25-NEW + 14-single-entity window (2026-08-15). The batch log line 复用 rss-feed-scan 已存分数 N 篇 is the tell that reuse-scores.json exists. Details: references/batch-tmp-artifact-score-recovery.md.
   3. Raw article frontmatter on disk: ^(?:vxc|score_vc):\s*(\d+) (often absent).
   4. : .

5. Publish date priority: raw frontmatter ^(?:source_published|publish_date|publish_time|published): → entity frontmatter publish_date: (verified: AML entity carries a real publish_date: 2026-08-12 while its raw has none) → body ^Published Time: → entity created: with ~ prefix (ingestion date, not original).

6. Report coverage: print VXC_COVERAGE=N/M — a low ratio tells you the batch window has unscored entities (normal for 39+-entity recovery rounds where log.md only records the top 5 in the 高价值 line).

Pitfalls (all verified 2026-08-14)

• grep-filtered log.md = 0 score facts. The 高价值 and domain-reject continuation lines carry no date. Always read the whole file.
• 高价值： prefix contaminates the norm key → 高价值重新审视交叉熵lmloss fails containment. Strip the prefix (re.sub(r'^.*?高价值[：:、]?\s*', '', line)) before regex-extracting pairs.
• Entity-slug-first, not raw-slug-first. entities/memory-for-llms-tsinghua-taxonomy-2026 has sources: with TWO raws: first raw (paper) has no score in log.md, second raw (mozhi reprint) matched RAW-tier vxc=35 — but the entity is NEW vxc=64. Same for stealing-reasoning-traces-proprietary-llm-apis (raw 30 vs entity 64). Match the entity's own basename against entry facts before any raw slug; a raw-tier score is the wrong answer for a NEW entity.
• %h vs %H hash mismatch silently breaks single-entity detection. Building the per-commit file-count map with --format=%H (full hash) but comparing against %h (short hash) from the subject line → every lookup misses → is_single always True → the FIRST vxc= in a multi-entity subject is assigned to every entity (handroid got 72 instead of 56; file-upload 72 instead of 49). Use %h in BOTH the map key and the comparison.
• Single-entity commit vxc formats vary: (vxc=56, NEW entity) (parens, after CJK text), — vxc=64 NEW entity + raw, InstEmb NEW(64). The vxc= regex alone misses NEW(64) style.
• Reject-line proxy titles are compressed and containment fails. 600倍加速视频生成 (56) vs entity title 600 倍加速 720p 视频实时生成 — 单卡 14B 模型推理优化; Linux内核漏洞 (64) vs 科维斯 AI（Corvus AI）— TencentOS 内核漏洞研究智能体首秀. Known-upgrade map (log.md 2026-08-13 domain-reject line → 08-14 expand/rescue entities): 600倍加速视频生成→video-generation-600x-acceleration-single-gpu-2026 (56), Linux内核漏洞→corvus-ai-tencentos-kernel-vulnerability-agent-2026 (64), COVERT→covert-vlmaas-covariant-obfuscation-eccv-2026 (56), Handroid→handroid-* (56, also in commit subject). Report these as vxc=NN (reject-line proxy).

Script

• scripts/extract_new_entities.py — the full validated extraction script (v5, 2026-08-14). Usage: python3 extract_new_entities.py [--since "24 hours ago"]. Prints TOTAL_NEW, VXC_COVERAGE, and a TSV sorted by vxc desc. Run from the wiki root (cd ~/wiki first — the script chdirs itself, but git must run inside the repo).
• Extend the OVERRIDES dict at the top for reject-line proxy scores (verified mappings only — see pitfalls).
• For batch-run /tmp artifact score recovery (tier 2 of the chain), see references/batch-tmp-artifact-score-recovery.md — includes the exact fname-keyed lookup snippet and artifact schema inventory.

Relationship to other skills

• wiki-status-report (user-owned, uneditable by curator): governs the report format and silent-or-deliver logic. This skill supplies the extraction engine. If wiki-status-report is adopted, merge this SKILL.md's pitfalls + script into it.
• wiki-auto-pipeline-recipes (user-owned): batch-pipeline debugging notes; adjacent territory.