wiki-maintenance

name	wiki-maintenance
description	Comprehensive wiki operations on the Obsidian wiki at ~/wiki — routine maintenance (frontmatter, tags, lint, orphans, git), content expansion (concepts, entities), batch quality sprints, and crisis repair (YAML corruption, citation gaps, duplicate deduplication).
category	wiki

Wiki Maintenance Playbook

Scope

Routine maintenance operations on the Obsidian wiki at ~/wiki (tech wiki) or ~/wiki-life (life wiki): frontmatter fixes, tag cleanup, lint validation, orphan detection, git housekeeping.

Prerequisites

# For tech wiki:
cd ~/wiki

# For life wiki:
cd ~/wiki-life

Lint Validation (always run after any change)

Tech wiki (~/wiki)

node scripts/wiki-lint.mjs .
# Expected: 0 error(s). Warnings are informational.

Life wiki (~/wiki-life)

No lint script exists — validation is manual via Python execute_code or shell inspection:

# Check frontmatter completeness
for f in entities/*.md concepts/*.md comparisons/*.md cases/*.md queries/*.md; do
  head -10 "$f" | grep -q "^type:" || echo "MISSING type: $f"
  head -10 "$f" | grep -q "^created:" || echo "MISSING created: $f"
  head -10 "$f" | grep -q "^updated:" || echo "MISSING updated: $f"
  head -10 "$f" | grep -q "^tags:" || echo "MISSING tags: $f"
done

# Check index.md stats vs actual
for d in entities concepts cases comparisons queries reviews "raw/articles"; do
  count=$(ls $d/*.md 2>/dev/null | wc -l | tr -d ' ')
  echo "$d: $count"
done

Skills Sync — Multi-Category rsync

Skills live in multiple category subdirectories under ~/.hermes/skills/: wiki/, creative/, devops/, software-development/, apple/, autonomous-ai-agents/, mcp/, etc. The wiki's skills/ directory is the union of all categories, synced via rsync.

Pitfall: Single-Category rsync + --delete Deletes Other Categories

Symptom: After syncing ~/.hermes/skills/wiki/ → ~/wiki/skills/ with --delete, skills like anti-ai-style-library, design-md, comfyui etc. disappear from ~/wiki/skills/ because they live in ~/.hermes/skills/creative/, not ~/.hermes/skills/wiki/.

Wrong (only syncs wiki-category skills, deletes everything else):

rsync -av --delete ~/.hermes/skills/wiki/ ~/wiki/skills/

Correct — sync ALL categories, one rsync per source dir without --delete (use --delete only on the first/primary source):

# Primary sync (wiki category, with --delete to remove stale files)
rsync -av --delete ~/.hermes/skills/wiki/ ~/wiki/skills/

# Additive syncs (other categories, NO --delete — they only add, never remove wiki-category files)
for cat in creative devops software-development apple autonomous-ai-agents mcp productivity research media data-science github note-taking smart-home social-media red-teaming mlops gaming email; do
  src="$HOME/.hermes/skills/$cat/"
  [ -d "$src" ] && rsync -av "$src" ~/wiki/skills/
done

Why no --delete on secondary categories: If creative skills sync with --delete, it would remove wiki-category files that don't exist in creative/. The --delete flag means "make the destination look exactly like the source" — removing anything not in the source.

Safer alternative — sync the parent directory directly (includes all categories):

# This copies ALL skill categories at once, --delete is safe here
rsync -av --delete ~/.hermes/skills/ ~/wiki/skills/ --exclude='*.pyc' --exclude='__pycache__'

Caveat: This also copies non-category files (e.g., DESCRIPTION.md at the top level). Verify with git status after sync.

Recovery if skills were deleted: Re-run the correct rsync command, then git add skills/ && git commit.

Nested .git Directories in Skills

Skill dirs under ~/.hermes/skills/ must NOT contain .git/ subdirectories — Git clients discover them and display phantom repos. See references/nested-git-repo-cleanup.md for diagnosis and fix.

Common Operations

Frontmatter Tag Cleanup

Problem: Files with tags: [] (empty array) or duplicate tags: YAML keys (last-value-wins — empty array overwrites real tags).

Detection:

import os, re
wiki_path = "entities"
empty = []; dup = []
for root, dirs, files in os.walk(wiki_path):
    for f in files:
        if not f.endswith('.md'): continue
        fp = os.path.join(root, f)
        with open(fp) as fh: content = fh.read()
        fm_match = re.search(r'^---\n(.*?)\n---', content, re.DOTALL)
        if not fm_match: continue
        fm = fm_match.group(1)
        if re.search(r'^tags:\s*\[\s*\]', fm, re.MULTILINE): empty.append(fp)
        if fm.count('tags:') > 1: dup.append(fp)

Fix empty arrays — keyword-based auto-tagging:

import re

def auto_tag(title, body, source):
    ti = title.lower(); bd = body.lower(); s = source.lower()
    kw_map = [
        ('nvidia', ['nvidia', 'gpu kernel', 'cuda']),
        ('claude-code', ['claude']),
        ('anthropic', ['anthropic']),
        ('security', ['security', 'hack', 'vulnerability', 'supply chain', 'exploit', 'malicious', 'phishing', 'fuzzing', 'cve']),
        ('agent', ['agent']),
        ('multi-agent', ['multi-agent', 'multiagent', 'multi agent']),
        ('harness-engineering', ['harness']),
        ('rag', ['rag', 'retrieval']),
        ('memory', ['memory']),
        ('mcp', ['mcp']),
        ('llm', ['llm', 'language model', '大模型', 'gpt', 'gemma', 'deepseek', 'minimax']),
        ('skill', ['skill']),
        ('openclaw', ['openclaw']),
        ('aws', ['aws', 'amazon', 'bedrock', 'sagemaker']),
        ('open-source', ['open-source', 'open source', '开源']),
        ('video', ['video']),
        ('inference', ['inference', '推理', 'speculative', 'kv cache']),
        ('evaluation', ['benchmark', '评估', '评测']),
        ('architecture', ['architecture', '架构']),
        ('engineering', ['engineering', '工程']),
        ('ai', ['ai', '人工智能']),
        ('news', ['newsletter', 'venturebeat', 'cio dive', 'theregister', 'hackernews']),
    ]
    t = []
    for tag, keywords in kw_map:
        for kw in keywords:
            if kw in ti or kw in bd: t.append(tag); break
    if not t:
        if 'wechat' in s or '微信' in ti: t = ['wechat', 'ai']
        elif 'newsletter' in s or 'rss' in s: t = ['newsletter', 'ai']
        else: t = ['article', 'ai']
    seen = set(); unique = []
    for x in t:
        if x not in seen: seen.add(x); unique.append(x)
    return unique[:5]

for fp in empty:
    with open(fp) as fh: content = fh.read()
    # Extract metadata, call auto_tag(), then:
    new_content = re.sub(r'^tags:\s*\[\s*\]', f"tags: [{', '.join(tags)}]", content, count=1, flags=re.MULTILINE)
    with open(fp, 'w') as fh: fh.write(new_content)

Patch Tool |- Prefix Corruption

After ANY patch tool edit (not just frontmatter), check the affected lines. The patch tool can corrupt YAML list item prefixes in BOTH frontmatter AND body lists (|- instead of - ).

Frontmatter (YAML parse failure):

sed -i '' 's/^|- /- /' entities/*.md

Body Related entities lists (visible corruption — multiple items merged into one line):

# Example: "- [[entities/A]]- [[entities/B]]- [[entities/C]]" merged into one list item
content = content.replace(
    '- [[entities/X]]- [[entities/Y]]- [[entities/Z]]',
    '- [[entities/X]]\n- [[entities/Y]]\n- [[entities/Z]]'
)

Always inspect list sections after patching Related entities or other - lists.

UTF-8 Byte Count vs Char Count — lint EXCESS INFERRED Inflation

Symptom: wiki-lint.mjs reports EXCESS INFERRED: entities/SLUG has N/N (100%) where N is wildly inflated (e.g., 368 for a 530-line file). Manual verification shows 0 citations, no actual self-links.

Root cause: wiki-lint.mjs reads files as raw bytes and uses byteLength / 50 to estimate paragraph count. Files containing emoji (🔥) or CJK Extension-B/C characters encode each character as 4 bytes instead of 3. This doubles the byte count and thus doubles the estimated paragraph count.

Detection:

import os
# Compare byte count vs char count for suspect files
fp = 'entities/十年老技术开发的-ai-agent-探索之路.md'
with open(fp, 'rb') as f: raw_bytes = len(f.read())
with open(fp, 'r', encoding='utf-8') as f: char_count = len(f.read())
ratio = raw_bytes / char_count
print(f"Byte/char ratio: {ratio:.2f}")  # ratio > 1.5 means heavy 4-byte chars
# Example: 36353 raw bytes / 18093 chars = 2.01 → INFLATED

Real-world case: entities/十年老技术开发的-ai-agent-探索之路.md — 36353 raw bytes / 18093 UTF-8 chars = 2.01 ratio. lint estimated 368 paragraphs, actual 0 citations in 530-line file. The file is clean; lint's estimate is wrong.

Rule: When EXCESS INFERRED count ≈ 2× file line count, suspect byte-count inflation. Verify with Python char count before assuming the file needs repair.

SHA-256 Computation for Raw Articles

Rule: SHA-256 must be computed on body content ONLY (strip YAML frontmatter), consistent with wiki-source-hasher.mjs.

import hashlib, re, os

def compute_sha256(filepath):
    with open(filepath, 'r', encoding='utf-8', errors='replace') as f:
        content = f.read()
    # Strip frontmatter
    fm_match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
    body = content[fm_match.end():] if fm_match else content
    return hashlib.sha256(body.encode('utf-8')).hexdigest()

Missing sha256 workflow: When lint flags raw articles missing sha256:

1. Compute SHA-256 from body (strip frontmatter)
2. If file has no frontmatter at all, add minimal frontmatter:

   ---
   source_url: <inferred or empty>
   ingested: 2026-05-01
   sha256: <computed>
   ---
   

3. If file already has frontmatter but no sha256, insert sha256: <computed> into existing frontmatter

Newsletter Typo — Quoted vs Unquoted newsletter in Tags

Symptom: 11 entity files had tags: ["newsletter"] (quoted string inside YAML array) instead of tags: [newsletter] (bare scalar). YAML treats "newsletter" as a quoted string, not a tag identifier.

Fix:

import re, os
for fn in os.listdir('entities'):
    if not fn.endswith('.md'): continue
    fp = f'entities/{fn}'
    with open(fp) as f: content = f.read()
    new = re.sub(r'tags:\s*\["newsletter"\]', 'tags: [newsletter]', content)
    if new != content:
        with open(fp, 'w') as f: f.write(new)

YAML Indentation Bug — tags: [] Empty But File Has Content

Symptom: 7 entity files report tags: [] (empty) in tag-normalization scan, yet the files exist and have body content. Running the auto-tagger on these files produces correct tags (e.g., ['claude-code', 'engineering']).

Root Cause (YAML spec): When tags: is at the same indentation level as sibling keys like review_value:, YAML interprets tags: as an empty list [] AND review_value: as a separate sibling key — not as tags: having the value review_value:. Python/YAML simply picks one (last-value-wins).

Example of broken frontmatter:

---
title: Some Entity
  tags:
  review_value: 7
  sources:

The indentation is ambiguous — tags: appears at the same level as review_value:. YAML parser treats tags: as an empty array and review_value: as a separate key.

Fix: Ensure tags: has its own indented value line:

tags: [claude-code, engineering]
review_value: 7
sources:

Detection: Both the empty-array check AND the duplicate-key check fire on this same bug pattern. Fix by ensuring tags: always has an inline array value on the same line, not a bare tags: followed by sibling keys.

Fix Duplicate Frontmatter Keys

When tags: appears 2+ times in YAML frontmatter, only the last value is used (YAML last-value-wins). Remove duplicates:

import re
for fp in dup_files:
    with open(fp) as fh: content = fh.read()
    fm_match = re.search(r'^---\n(.*?)\n---', content, re.DOTALL)
    fm = fm_match.group(1)
    lines = fm.split('\n')
    seen_tags = False; new_lines = []
    for line in lines:
        if re.match(r'^tags:', line):
            if not seen_tags: new_lines.append(line); seen_tags = True
            # Skip subsequent tags: lines
        else:
            new_lines.append(line)
    new_content = content.replace(fm_match.group(1), '\n'.join(new_lines))
    with open(fp, 'w') as fh: fh.write(new_content)

Git Housekeeping Before Large Commits

Rule: Always git stash before starting bulk maintenance to get a clean state.

Nested .git directories in skills/: Skill dirs under ~/.hermes/skills/ must NOT contain .git/ subdirectories — Git clients discover them and display phantom repos. Delete .git/ from any skill that was cloned from GitHub. See references/nested-git-repo-cleanup.md for diagnosis and fix.

Git commit blocked by deleted tracked files: If tracked files were deleted but not staged, git add entities/*.md alone is blocked:

# Unstage deletions first, then stage what you want
git add -u _archive/raw-duplicates/   # clear deleted-file staging
git add entities/*.md
git commit -m "fix: ..."

Orphan Detection

After any linking phase, check for true orphans (0 in-links AND 0 out-links):

node scripts/wiki-lint.mjs . 2>&1 | grep -A5 "True orphans"

• entities/ orphans = actionable (need links)
• raw/ orphans = expected (source material layer, design intent)

Orphan Recovery — Batch Tag+Keyword Matching

When orphan count is large (30+), manual per-entity linking doesn't scale. Use automated tag+keyword matching:

Algorithm

1. Build slug_tags dict from frontmatter (use yaml.safe_load — NOT regex — to handle both inline tags: [a, b] and block tags:\n - a formats)
2. Build slug_title dict from frontmatter titles
3. Build tag_index: defaultdict(set) mapping each tag → set of slugs
4. For each orphan, compute candidate score = tag_overlap * 2 + title_keyword_overlap
5. Add ## 相关实体 section with top 2-3 verified candidates to the orphan
6. Also add backlinks from candidate entities to the orphan (bidirectional linking)

from collections import Counter, defaultdict

# Build tag index
tag_index = defaultdict(set)
for slug, tags in slug_tags.items():
    for t in tags:
        tag_index[t].add(slug)

# Score candidates for each orphan
candidates = Counter()
for t in my_tags:
    for other in tag_index[t]:
        if other != orphan:
            candidates[other] += 2  # tag match weight

# Title keyword overlap
title_words = set(re.findall(r'[a-z]{3,}', my_title.lower()))
for other_slug, other_title in slug_title.items():
    if other_slug == orphan: continue
    other_words = set(re.findall(r'[a-z]{3,}', other_title.lower()))
    overlap = title_words & other_words
    if len(overlap) >= 2:
        candidates[other_slug] += len(overlap)

top = [s for s, _ in candidates.most_common(5) if s in entity_slugs]

Insertion Pattern

• Add ## 相关实体 section BEFORE the → [[raw/articles/...|原文存档]] backlink line
• If that section already exists, append new links to it
• Always bump updated: date
• Always verify targets exist on disk before adding wikilinks

Pitfall: YAML Tag Format — Inline vs Block

The regex r'^tags:\s*\[([^\]]*)\]' only matches inline format tags: [a, b]. Many entity files use block format:

tags:
  - claude-code
  - anthropic
  - security-incident

Always use yaml.safe_load() for tag parsing, not regex. The FM_RE in wiki-lint.mjs captures the frontmatter block; pass it through PyYAML:

import yaml
fm_match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
if fm_match:
    fm = yaml.safe_load(fm_match.group(1))
    tags = fm.get('tags', [])  # works for both inline and block

Pitfall: Quoted Tags in YAML

Some files have tags: ["claude-code", "anthropic"] (quoted strings) instead of tags: [claude-code, anthropic]. While YAML accepts both, the quoted form causes tag-match failures because the tag index stores "claude-code" (with quotes) as the key. Fix:

# Remove quotes from tags
content = re.sub(r'tags:\s*\[([^\]]*)\]', 
    lambda m: 'tags: [' + re.sub(r'"([^"]+)"', r'\1', m.group(1)) + ']',
    content)

Results (2026-06-17 session)

• 35 true orphans → 1 (97% reduction)
• 304 zero-inbound entities → 6 (98% reduction)
• 507 backlinks added across 317 entity files
• 38 files fixed for quoted tags
• 8 files auto-tagged via keyword rules
• Lint: 0 errors

See references/orphan-recovery-tag-matching-2026-06-17.md for full session transcript.

Index Maintenance

If new entities were created but not added to index.md:

node scripts/wiki-lint.mjs . 2>&1 | grep "MISSING"
# Add missing entries to index.md, then commit separately

MOC Index Maintenance

MOC (Map of Content) files live in moc/ and MUST be listed in index.md under the ## MOCs section. When creating a new MOC, immediately add its index entry:

# Check if MOC is missing from index
import re
with open('index.md') as f: content = f.read()
moc_section = re.search(r'## MOCs?\n(.*?)(?=\n## |\Z)', content, re.DOTALL)
if moc_section:
    moc_entries = re.findall(r'\[\[moc/([^\]|]+)', moc_section.group(1))
    import os
    for f in os.listdir('moc'):
        if f.endswith('.md') and f[:-3] not in moc_entries:
            print(f"MISSING from index: moc/{f}")

Insert format: - [[moc/slug|Display Title]] — Brief description

MOC Table Wikilink Escaping (Critical Pitfall)

When a MOC contains a markdown table with [[wikilink|display text]] in a cell, the | inside the wikilink is parsed as a table column separator, not a wikilink display-text delimiter. This breaks the table layout.

Fix: Escape the pipe inside the wikilink with \|:

| Source | Size |
|--------|------|
| [[raw/articles/slug\|Display Text]] | 12KB |

Linter false positive: The wiki-lint.mjs regex WL_RE does NOT understand \| escaping — it reports BROKEN LINK: moc/X -> [[raw/articles/slug\]] (treating the \ as part of the slug). This is a linter limitation, not a real broken link. The wikilink renders correctly in Obsidian. Use git commit --no-verify to bypass.

Verification: Check with Python byte-level inspection:

with open('moc/file.md', 'rb') as f:
    lines = f.read().split(b'\n')
for i, line in enumerate(lines):
    for j in range(len(line)):
        if line[j:j+1] == b'|' and j > 0 and line[j-1] == 0x5c:
            print(f'Line {i+1}: escaped pipe at byte {j}')

Dangling Wikilink — Wrong Directory Prefix

When lint reports BROKEN LINK: entities/X -> [[entities/Y]] and no entities/Y.md exists, check if the target actually lives in a different directory:

• entities/agent-self-improvement-loops → file is at concepts/agent-self-improvement-loops.md → fix: [[concepts/agent-self-improvement-loops]]
• entities/anthropic-cache-tokenomics → no entity exists, only raw article → fix: [[raw/articles/anthropic_cache_tokenomics]]

Fix pattern:

# Fix dangling entities/ → concepts/ or raw/articles/
broken_map = {
    'entities/agent-self-improvement-loops': 'concepts/agent-self-improvement-loops',
    'entities/anthropic-cache-tokenomics': 'raw/articles/anthropic_cache_tokenomics',
}
for fname in os.listdir('entities'):
    fp = f'entities/{fname}'
    with open(fp) as f: content = f.read()
    for old, new in broken_map.items():
        content = content.replace(f'[[{old}', f'[[{new}')
    with open(fp, 'w') as f: f.write(content)

Wiki-Lint Citation Format (Critical)

The only citation format wiki-lint.mjs recognizes is ^[raw/articles/SLUG.md] — NOT [[raw/articles/SLUG.md]] or [[raw/articles/SLUG.md|原文存档]].

This is enforced by CITATION_SIMPLE_RE in wiki-lint.mjs:

const CITATION_SIMPLE_RE = /\^\[([^\]]*\.md)\]/;  // NO /g flag

The ^ is a literal caret character, not a regex anchor — the lint system uses this to mark source-attributed paragraphs.

Insertion rules:

• Only prose lines (non-header #, non-list - /* , non-blockquote > , non-empty) get citations appended
• Must skip frontmatter lines (including the closing ---) — the closing --- line is NOT a frontmatter boundary marker in the body; it must be excluded by line-index comparison
• Citation must end with .md] — e.g. ^[raw/articles/slug.md], not ^[raw/articles/slug] (lint's MALFORMED CITATION check requires .md suffix)
• Do NOT use [[...]] — this is the Obsidian display format, not the lint citation format

Frontmatter source normalization: Many entities have sources: [raw/articles/slug] (no .md). The frontmatter value is for reference; the citation added to body must use the .md extension by consulting the filesystem (raw/articles/SLUG.md exists with extension).

Nested [[ in frontmatter sources: Some sources: YAML values contain [[...]] (e.g. sources: [[factory-missions-multi-agent-architecture]]). When appending ^[raw/articles/[[...]]].md the nested brackets corrupt the citation. Parse frontmatter sources carefully — strip any [[/]] display wrappers before constructing the citation.

rawSourceMap key structure: The output of find-raw-sources.mjs (/tmp/raw_source_map.json) uses numeric string keys ("0", "1"...), not filenames. The actual filename is in entry.file. Build a filename → source map manually:

const fileMap = {};
Object.values(rawSourceMap).forEach(entry => {
  const key = entry.file.replace(/\.md$/,'');
  fileMap[key] = entry.source;
});

Malformed updated: Fields

Detection: updated: values not matching YYYY-MM-DD (e.g., type: article, "2026-05-14" with quotes).

import re, os
bad = []
for fn in os.listdir('entities'):
    if not fn.endswith('.md'): continue
    with open(f'entities/{fn}') as f:
        c = f.read(2000)
    m = re.search(r'^updated:\s*(.+)$', c, re.MULTILINE)
    if m and not re.match(r'^\d{4}-\d{2}-\d{2}$', m.group(1).strip()):
        bad.append((fn, m.group(1).strip()))
print(f'Bad updated: {len(bad)}')
for fn, val in bad[:5]: print(f'  {fn}: {val}')

Anomalous review_value (>10)

Detection: Values like 60, 64, 72 indicate data entry error (scale is 1–10).

grep "review_value: [6-9][0-9]\|review_value: [1-9][0-9][0-9]" entities/*.md

Duplicate tags: YAML Keys — FIRST Non-Empty Wins, Not Last

Symptom: 100 raw files have tags: appearing 2-4 times in frontmatter (e.g., first batch-inserted tags, then empty tags: [] overwrite). YAML parser uses last-value-wins, so empty [] overwrites correct tags, leaving files with tags: [] despite earlier correct values.

Detection:

import re, os
dup = []
for fn in os.listdir('raw/articles'):
    if not fn.endswith('.md'): continue
    fp = f'raw/articles/{fn}'
    with open(fp) as f: raw = f.read()
    m = re.match(r'^---\n(.*?)\n---\n', raw, re.DOTALL)
    if not m: continue
    tags_lines = [l for l in m.group(1).split('\n') if l.startswith('tags:')]
    if len(tags_lines) > 1:
        dup.append((fn, len(tags_lines)))
print(f"Files with duplicate tags: keys: {len(dup)}")

Fix: Keep the FIRST non-empty tags line, remove all duplicates. NOT the last (which may be empty []).

for fn, count in dup:
    fp = f'raw/articles/{fn}'
    with open(fp) as f: raw = f.read()
    m = re.match(r'^---\n(.*?)\n---\n', raw, re.DOTALL)
    fm_text = m.group(1)
    
    # Find first non-empty tags line
    first_tags = None
    for line in fm_text.split('\n'):
        if line.startswith('tags:') and '[]' not in line:
            first_tags = line
            break
    
    if first_tags:
        new_fm_lines = [l for l in fm_text.split('\n') if not l.startswith('tags:')]
        new_fm_lines.append(first_tags)
        new_fm_str = '\n'.join(new_fm_lines)
        new_raw = raw[:m.start()] + '---\n' + new_fm_str + '\n---\n' + raw[m.end():]
        with open(fp, 'w') as f: f.write(new_raw)

Rule: When inserting tags: into frontmatter that may already have one, delete existing tags: lines first, then append the correct one. Do NOT add a second tags: line without removing the old one.

Domain → Tag White list for Empty-Tag Raw Articles

Problem: ~790 raw articles have tags: [] (empty). Source URL domain can auto-infer tags.

White list (build from existing tagged articles + manual additions):

DOMAIN_TAGS = {
    'mp.weixin.qq.com': ['wechat', 'article', 'claude', 'openai'],
    'aws.amazon.com': ['aws-china-blog', 'agentic-ai'],
    'developer.nvidia.com': ['nvidia', 'inference'],
    'www.anthropic.com': ['anthropic', 'claude'],
    'arxiv.org': ['arxiv'],
    'stochasticparrot.substack.com': ['newsletter', 'llm'],
    'news.ycombinator.com': ['hackernews'],
    'substack.com': ['newsletter'],
    'medium.com': ['article'],
    'deepmind.google': ['deepmind', 'ai'],
    'huggingface.co': ['huggingface'],
    'github.com': ['github'],
    'juejin.cn': ['juejin'],
    'thehackernews.com': ['security'],
    # ... extend as needed
}

Batch apply: substring match (domain contains key OR key contains domain). Round 1 fixed 437, round 2 fixed 185 (622 total). ~168 remain for niche domains — add manually or skip as low priority.

Tag Synonym Consolidation

Problem: Inconsistent tag naming inflates unique tag count (2227 unique, 1216 used once).

Consolidation map:

CONSISTENCY_MAP = {
    'Claude-Code': 'claude-code', 'Anthropic': 'anthropic', 'Agent': 'agent',
    'Harness': 'harness', 'OpenClaw': 'openclaw', 'AI-agents': 'ai-agent',
    'design-systems': 'design-system', 'eval': 'evals', 'tools': 'tool',
    'hooks': 'hook', 'lowcode': 'low-code', 'design-engineering': 'design-system',
    'component-architecture': 'architecture', 'modular-architecture': 'architecture',
    'state-management': 'state-machine', 'context-construction': 'context-engineering',
    'agentic-rag': 'rag', 'classic-rag': 'rag', 'browser-use': 'computer-use',
    'amazon': 'aws', 'deepseek-v4': 'deepseek', 'autogen': 'multi-agent',
    'dify': 'agent', 'agentscope': 'agent', 'letta': 'memory',
    'hci': 'ux-design', 'ux': 'ux-design', 'zeroheight': 'design-system',
    'openai-codex': 'codex', 'chatgpt': 'gpt',
}

Apply and deduplicate resulting array. Verify with yaml.safe_load() per batch.

updated Backfill from created

When updated field missing but created exists, copy value:

for fn in os.listdir('entities'):
    fp = f'entities/{fn}'
    with open(fp) as f: raw = f.read()
    m = re.match(r'^---\n(.*?)\n---\n', raw, re.DOTALL)
    fm = yaml.safe_load(m.group(1)) if m else {}
    if 'updated' not in fm and fm.get('created'):
        old_fm = m.group(1)
        new_fm_lines = []
        for line in old_fm.split('\n'):
            if line.startswith('created:'):
                new_fm_lines.append(line)
                new_fm_lines.append(line.replace('created:', 'updated:'))
            else:
                new_fm_lines.append(line)
        new_raw = raw[:m.start()] + '---\n' + '\n'.join(new_fm_lines) + '\n---\n' + raw[m.end():]
        with open(fp, 'w') as f: f.write(new_raw)

P0→P1→P2 Systematic Quality Sprint (2026-05-23)

P0→P1→P2 Systematic Quality Sprint (2026-05-23)

Session pattern: run lint → classify all warnings by type → assign P0/P1/P2 → fix in priority order.

重要发现 (2026-05-23 更正): 之前 wikilink 跨文件解析 bug（未 strip entities/ 前缀 + regex [^|\\]] 错误）导致统计错误。之前报告的 28 条 entity→entity links 和 96% orphan 率是 bug 导致的假象。

真实数据（2026-05-23 验证）:

• entity→entity wikilinks: 4703条（shell grep 与 Python 一致）
• Orphan entities（0 in-links）: 870/1673（52%）— 所有 entity 都是其他 entity 的 outlink 目标
• Hubs（≥3 in-links）: 439个，Top 1 hub 被引用 47 次
• Cross-validate: grep -oE '\\[\\[entities/[^]]+\\]\\]' entities/ | wc -l → 4595 ≈ Python sum(in_counts) 4703 ✓ 正确统计方法: 见 wiki-audit skill — 必须 extract_target() 去除 entities/ 前缀，且 source 只遍历 entities/ 目录。

P0（必须立即修）:

• YAML frontmatter corruption（导致 wiki-lint.mjs 报错）
  • |- 前缀污染：patch tool 插入 |- 而非 - → 修复: sed -i '' 's/^|- /- /' entities/*.md
  • tags: [...] 内联数组 + 缩进块序列混用（9个文件，2026-05-25）: YAML 解析器无法处理同一 tags: 字段既有内联数组值又有缩进列表项
    • Pattern: tags: [item1, item2]\n - item1\n - item2 — 内联数组值存在，缩进项被解析为悬空块序列
    • 修复: 删除内联 tags: [...] 行，保留缩进块序列项转换为内联格式 tags: [item1, item2]
    • 检测: re.search(r'^tags:\s*\n\s+- ', fm, re.MULTILINE) 或 re.search(r'^tags:\s*\[\s*\]\s*\n\s+- ', fm, re.MULTILINE)
    • 验证: yaml.safe_load() 通过 → 0 errors
• Design-intent duplicates（1042 raw↔entity title 一致 = 正确，跳过）

P1（高优先级）:

• Citation 批量补全（2026-05-25）: 420个 entity 有 sources: [raw/articles/SLUG] 但 body 无 ^[raw/articles/SLUG.md] 引用
  • 策略: 找到第一段正文（非 #/```/-/>/），追加 ^[raw/articles/SLUG.md]
  • 前置条件: 检查 sources: 只有1个 raw source，body 无 citation
  • 批量修复: for entity in candidates: txt[:pos] + citation + txt[pos:]
  • 结果: 34.9% → 59.9% coverage（+25pp，commit 6f6eb927）
  • 39 thin entities（<50 words）：全部有 ## 深度分析，是真实内容，无需修复
  • 794 raw empty tags：同一任务拆为"诊断"和"执行"两步，实际是批量 tag 补全（从 source_url 推断 domain）
  • 198 entity 缺 updated（171 无从追溯，27 可从 created 回填）

P2（中优先级）:

• 64 raw thin <100 words
• Tag taxonomy consolidation（白名单机制）
• Orphan 分析：699个 orphan 有 out-links，引用 hub 但无反向链接 — 当前结构可接受，非优先修复

Git commit 节奏: 每类修复单独 commit，不要等全部完成再统一提交。

Git Stash 冲突标记批量清理（2026-06-10 实战）

症状: git stash pop 或 merge 后，大量 entity 文件残留 <<<<<<< Updated upstream / ======= / >>>>>>> Stashed changes 冲突标记。YAML 解析器无法处理，lint 报数百 errors。

检测:

cd ~/wiki/entities && grep -l "<<<<<<< Updated upstream" *.md | wc -l

修复（Python 批量，选择保留 upstream 版本）:

import os, re
entities_dir = "/Users/jinguo/wiki/entities"
fixed = 0
for fname in os.listdir(entities_dir):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    if '<<<<<<< Updated upstream' not in content:
        continue
    new_content = re.sub(
        r'<<<<<<< Updated upstream\n(.*?)=======\n(.*?)>>>>>>> Stashed changes\n',
        lambda m: m.group(1),  # 保留 upstream
        content, flags=re.DOTALL
    )
    if new_content != content:
        with open(fpath, 'w', encoding='utf-8') as f:
            f.write(new_content)
        fixed += 1
print(f"Fixed {fixed} files")

选择策略: 先抽查 3-5 个文件确认 upstream vs stashed 哪个更完整（stashed 可能含更新的 summary/links），再决定 m.group(1) 还是 m.group(2)。

验证: 修复后必须跑 node scripts/wiki-lint.mjs . 确认 errors 大幅下降。2026-06-10 实测：1005 文件修复，errors 681→27。

Life Wiki (~/wiki-life) Audit Patterns

wiki-life is a separate Obsidian vault at ~/wiki-life for personal development content (自律/职业/心理/关系/认知). It has a different structure and no lint script.

Directory Structure (life wiki)

wiki-life/
├── entities/        # 12 deep-dive topic pages (e.g., 自律系统, 职业发展系统)
├── concepts/        # 4 core concept definitions (复利效应, 系统思维, etc.)
├── cases/           # 6 case studies (职业转型, 冲突修复, etc.)
├── comparisons/     # 5 method comparisons (GTD vs 时间块, 买房vs租房)
├── queries/         # 7 navigation/guide pages (Life Dashboard, Agent面试准备)
├── reviews/         # 2 monthly review templates
├── raw/articles/    # 10 source articles
├── index.md         # Master index (must be kept in sync)
├── log.md           # Operation history
├── AGENTS.md        # Agent instructions (frontmatter schema, scoring thresholds)
└── QUALITY.md       # Content quality criteria (7 reject types, 5 quality dimensions)

Common wiki-life Issues

1. Index.md stats drift — index.md claims outdated page counts. Fix:

import os
wiki = "/Users/jinguo/wiki-life"
for d in ['entities', 'concepts', 'cases', 'comparisons', 'queries', 'reviews', 'raw/articles']:
    p = os.path.join(wiki, d)
    count = len([f for f in os.listdir(p) if f.endswith('.md')]) if os.path.isdir(p) else 0
    print(f"{d}: {count}")

Then update the ## Stats section in index.md and the Last updated date.

2. Broken wikilinks in index.md — index.md may link to concepts/entities that don't exist yet (e.g., [[concepts/habit-loop]], [[queries/job-search-toolkit]]). These are planned pages that were never created. Fix: convert to plain text (remove [[...]] brackets, keep display text).

3. log.md corruption — + prefix lines (diff artifact) can appear after manual edits. Fix: strip leading + and trailing --- separators.

4. Missing frontmatter fields — some pages lack confidence and provenance_state. The expected schema per AGENTS.md:

---
title: "中文标题"
created: YYYY-MM-DD
updated: YYYY-MM-DD
type: entity|concept|comparison|case|query|guide|strategy
tags: [tag1, tag2]
confidence: 0.85       # optional but recommended
provenance_state: "extracted"|"merged"  # optional but recommended
---

5. Generated report files not gitignored — scripts/inbox_report_*.md should be in .gitignore.

6. Broken citations are DESIGN INTENT — wiki-life entities are synthesized overviews citing books/papers (e.g., "Getting Things Done", "Why We Sleep") that don't have raw article files. This is NOT a bug; it's the wiki-life design pattern. Do NOT remove them.

7. Batch frontmatter fix — --- section dividers cause duplicate metadata — When using content.replace('---', ...) to insert confidence/provenance_state before the closing ---, files with body section dividers (e.g., life-dashboard.md uses --- as visual separators between sections) will fire the replacement on EVERY ---, producing N copies of the metadata in the body. Fix: use re.match(r'^---\n.*?\n---\n', content, re.DOTALL) to isolate the first frontmatter block, then modify only that block. Or use patch tool targeting the specific frontmatter line. Detection: check for confidence: or provenance_state: appearing in the body (after the first ---).

Comprehensive Health Check Workflow (wiki-life)

When asked to "全面体检" (full health check) on wiki-life, follow this systematic approach:

Phase 1: Explore — understand the current state:

cd ~/wiki-life
# Directory structure
for d in entities concepts cases comparisons queries reviews "raw/articles"; do
  echo "$d: $(ls $d/*.md 2>/dev/null | wc -l) files"
done
# Git status
git status --short
git log --oneline -5
# Read ground-truth docs
head -50 AGENTS.md

Phase 2: Audit — run Python execute_code to find all issues at once:

import os, re
wiki = "/Users/jinguo/wiki-life"

# 2a. Build existing file sets
raw_dir = os.path.join(wiki, "raw/articles")
existing_raw = {f for f in os.listdir(raw_dir) if f.endswith('.md')}
all_slugs = set()
for d in ['entities','concepts','cases','comparisons','queries','raw/articles','templates','reviews']:
    for f in os.listdir(os.path.join(wiki,d)):
        if f.endswith('.md'): all_slugs.add(f[:-3])

# 2b. Broken citations
for d in ['entities','concepts','cases','comparisons','queries']:
    for f in os.listdir(os.path.join(wiki,d)):
        if not f.endswith('.md'): continue
        for c in re.findall(r'\^\[raw/articles/([^\]]+)\]', open(os.path.join(wiki,d,f)).read()):
            if c not in existing_raw: print(f"BROKEN CITATION: {d}/{f} -> {c}")

# 2c. Broken wikilinks
for d in ['entities','concepts','cases','comparisons','queries']:
    for f in os.listdir(os.path.join(wiki,d)):
        if not f.endswith('.md'): continue
        for wl in re.findall(r'\[\[([^\]]+)\]\]', open(os.path.join(wiki,d,f)).read()):
            t = wl.split('|')[0] if '|' in wl else wl
            if t.startswith('http') or t.startswith('#'): continue
            slug = t.split('/')[-1] if '/' in t else t
            if slug and slug not in all_slugs: print(f"BROKEN WL: {d}/{f} -> [[{t}]]")

# 2d. Duplicate confidence/provenance in body (batch-fix artifact)
for d in ['entities','concepts','cases','comparisons','queries']:
    for f in os.listdir(os.path.join(wiki,d)):
        if not f.endswith('.md'): continue
        c = open(os.path.join(wiki,d,f)).read()
        m = re.match(r'^---\n.*?\n---\n', c, re.DOTALL)
        if m and 'confidence:' in c[m.end()]: print(f"DUP CONF: {d}/{f}")

# 2e. [[index.md]] pattern (should be [[index]])
for d in ['entities','concepts','cases','comparisons','queries']:
    for f in os.listdir(os.path.join(wiki,d)):
        if not f.endswith('.md'): continue
        if '[[index.md' in open(os.path.join(wiki,d,f)).read(): print(f"INDEX.MD: {d}/{f}")

# 2f. Frontmatter completeness
for d in ['entities','concepts','cases','comparisons','queries']:
    for f in os.listdir(os.path.join(wiki,d)):
        if not f.endswith('.md'): continue
        c = open(os.path.join(wiki,d,f)).read()
        m = re.match(r'^---\n(.*?)\n---', c, re.DOTALL)
        if not m: print(f"NO FM: {d}/{f}"); continue
        for field in ['title','created','updated','type','tags']:
            if field not in m.group(1): print(f"MISSING {field}: {d}/{f}")

# 2g. Index.md stats vs actual
for d in ['entities','concepts','cases','comparisons','queries','reviews','raw/articles']:
    p = os.path.join(wiki, d)
    count = len([f for f in os.listdir(p) if f.endswith('.md')]) if os.path.isdir(p) else 0
    print(f"  {d}: {count}")

Phase 3: Fix in priority order:

• P0: log.md corruption, index.md stats, broken frontmatter
• P1: broken citations (remove), broken wikilinks (convert to plain text), missing fields
• P2: duplicate metadata artifacts, .gitignore patterns

Phase 4: Verify — re-run the audit script and confirm all counts are 0.

Phase 5: Commit — one commit per logical group, with descriptive messages.

wiki-life Scoring Threshold (from AGENTS.md)

Metric	Threshold	Notes
Value × Confidence	≥ 45	(vs 49 in tech wiki)
Actionability	独特方法/工具/模板可破格
Personal insight	反常识经验、真实案例可破格

7 Reject Types (life wiki only)

鸡汤文、毒鸡汤、软文/营销文、营销号、标题党、投射文、情绪包 — see QUALITY.md for full criteria.

YAML Frontmatter Corruption — 36-Entity批量修复（2026-05-23）

根因: 标题含 ASCII 双引号 "（非中文引号），被嵌入 YAML 双引号字符串内部导致解析冲突。

三种损坏模式:

# Pattern A: dquote 内嵌 inner quotes
title: "Anthropic联创：2028年实现"AI自我构建"的概率超过60%"
# Pattern B: smushed key（值紧贴引号，无空格分隔）
title: "harness-engineering实践"description: ...
# Pattern C: unquoted title 含冒号
title: Claude Code: GitHub CEO 评价 AI 编码趋势

修复策略（混用 v4+v5 函数）:

def fix_title_quotes(title):
    # 修复 Pattern A: "AI" → \"AI\"
    t = re.sub(r'(?<=[\"\'])(\w[\"\'] (?=[\"\']|$))', lambda m: '\\"' + m.group(1).replace('"', '\\"'), title)
    # 修复 Pattern B: "实践"description → "实践"\ndescription:
    t = re.sub(r'([\"\'}])\s*([a-z_]+:)', r'\1"\n\2', t)
    return t

def fix_smushed_keys(content):
    lines = content.split('\n')
    result = []
    for line in lines:
        m = re.match(r'^(\s*)(\S+):\s*["\'](.+)["\']+([a-z_]+:.*)$', line)
        if m:
            result.append(m.group(1) + m.group(2) + ': "' + m.group(3) + '"')
            result.append(m.group(1) + m.group(4))
        else:
            result.append(line)
    return '\n'.join(result)

def fix_unquoted_colon(content):
    lines = content.split('\n')
    for i, line in enumerate(lines):
        m = re.match(r'^(\s*title:\s*)(.+\S.*[：:].+\S.*)$', line)
        if m and not line.startswith('#'):
            lines[i] = m.group(1) + '"' + m.group(2).replace('"', '\\"') + '"'
    return '\n'.join(lines)

验证: 所有 36 个文件用 yaml.safe_load() 验证通过后提交。

修复后 lint: 0 errors, 1494 warnings（从 1499 减少 5）。

Patch tool |- 前缀污染：frontmatter 列表项经 patch 后会被插入 |- 而非 - ，导致 YAML 解析失败。修复：

sed -i '' 's/^|- /- /' entities/*.md

每次 patch frontmatter 后都要运行此修复。

YAML Tags Block Corruption — Orphaned List Items（2026-05-25）

两种损坏模式（均因 YAML block sequence 缩进解析冲突）：

Pattern 1 — tags: 空 key 后跟缩进列表（67个文件）：

tags:
  - harness
  - llm-agent
  - production

tags: 作为空标量解析，后面 - 缩进项变成悬空 block sequence。

Pattern 2 — tags: [] 空数组后跟缩进列表（18个文件）：

tags: []
  - default
title: "Yum Brands' tech chief on building its AI backbone"

tags: [] 后遇到缩进的 - default 导致 YAML 解析器状态混乱（block sequence 内嵌在 flow sequence 后）。

检测：

import re, os
entities_dir = "/Users/jinguo/wiki/entities"
bad = []
for fname in os.listdir(entities_dir):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    if re.search(r'^tags:\s*\n\s+- ', content, re.MULTILINE):
        bad.append((fname, 'empty_key'))
    elif re.search(r'^tags:\s*\[\s*\]\s*\n\s+- ', content, re.MULTILINE):
        bad.append((fname, 'empty_list'))

print(f"Pattern 1 (tags:\\n  -): {sum(1 for _,t in bad if t == 'empty_key')}")
print(f"Pattern 2 (tags: []\\n  -): {sum(1 for _,t in bad if t == 'empty_list')}")

修复（两种 pattern 统一处理）：

import re, os

entities_dir = "/Users/jinguo/wiki/entities"

for fname in os.listdir(entities_dir):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    
    # 跳过无损坏的文件
    if not (re.search(r'^tags:\s*\n\s+- ', content, re.MULTILINE) or
            re.search(r'^tags:\s*\[\s*\]\s*\n\s+- ', content, re.MULTILINE)):
        continue
    
    def fix_tags_block(m):
        block = m.group(0)
        lines = block.split('\n')
        items = []
        for line in lines[1:]:  # skip "tags:" or "tags: []"
            mo = re.match(r'^  - (.+)', line)
            if mo:
                items.append(mo.group(1))
            elif line.strip() == '':
                break
            else:
                break
        if items:
            return "tags: [" + ", ".join(items) + "]\n"
        else:
            return "tags: []\n"
    
    new_content = re.sub(
        r'(?:tags:\n|tags: \[\]\n)(?:  - [^\n]*\n)+',
        fix_tags_block,
        content
    )
    
    if new_content != content:
        with open(fpath, 'w', encoding='utf-8') as f:
            f.write(new_content)

# 验证
import yaml
for fname in os.listdir(entities_dir):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    # lint 用 FM_RE 非贪婪匹配，取 frontmatter 到第一个 "\n---" 为止
    # 如果 body 有 table separator "---"，frontmatter 边界在第一个 "\n---"（即 closing ---）
    # readFrontmatter() 做简单 split 而非 yaml.safe_load()，所以只验证 key: 格式
    m = re.match(r'^---\n([\s\S]*?)\n---', content)
    if m:
        fm_lines = m.group(1).split('\n')
        for line in fm_lines:
            if ':' in line:
                key = line[:line.index(':')].strip()
                if key and key not in ('tags', 'title', 'type', 'source', 'source_url',
                                      'review_value', 'review_confidence', 'review_recommendation',
                                      'review_stars', 'ingested', 'created', 'updated', 'published',
                                      'description', 'summary', 'sha256'):
                    pass  # 非标准 key 可能需检查

关键发现 — lint script FM_RE 行为：FM_RE = /^---\n([\s\S]*?)\n---/ 是非贪婪匹配，在第一个 \n--- 处停止。Body 中的 --- table separator 出现在 frontmatter closing --- 之后，不会干扰边界检测。Pattern 2 文件的 74 个 NO FRONTMATTER 报告是暂存变更（未 commit）导致的假象，commit 后 lint 恢复 0 errors。

修复后 lint：0 errors, 1491 warnings（commit b9d66366）。 每次 patch frontmatter 后都要运行此修复。

Cron Job Self-Audit Pattern

When reviewing whether cron job schedules need adjustment, the right signals are:

Inbox volumes (truthful indicator of consumption rate):

RSS inbox:      0  → 消费跟得上，频率 OK
WeChat inbox:   0  → 同上
Newsletter:    ~47 → 生产=消费，频率 OK

Heartbeat files (~/wiki/heartbeat/*.last-run) show actual last-run timestamps:

for f in heartbeat/*.last-run; do echo "$f: $(cat $f)"; done

Git commit pattern confirms active processing:

git log --oneline --since="2 days ago" -- raw/rss-inbox/ raw/wechat-inbox/

Don't just trust the schedule — a job with every 20m that hasn't run in 8h is effectively paused.

Script Output Parsing Failures — execSync ENOBUFS

Symptom: execSync via child_process throws ENOBUFS error when a script's output exceeds pipe buffer (~1MB on macOS).

Example: wiki-contradiction-scan.py outputs 988K+ chars → execSync buffer overflow → output truncated → downstream parsing fails silently → reports "scan failed" despite script working fine.

Fix: Pipe output through tail -c N to limit bytes before passing to Node.js:

// WRONG — ENOBUFS on large outputs:
const raw = sh("python3 scripts/wiki-contradiction-scan.py 2>&1");

// CORRECT — cap at 5KB, parse succeeds:
const raw = sh("python3 scripts/wiki-contradiction-scan.py 2>&1 | tail -c 5120");

Where stats are determines head vs tail: If summary stats are at the END of output, use tail -c N. If at the START, use head -c N.

Parsing robustness: Use trim().startsWith() for whitespace-safe line matching:

// WRONG — fragile with variable leading whitespace:
.filter(l => l.startsWith('  Entities with'))

// CORRECT — trim first, then match:
.filter(l => l.trim().startsWith('Entities with'))

Contradiction-Scan Algorithm Rewrite (2026-05-23)

Problem: wiki-contradiction-scan.py was generating 25,197 "potential contradictions" — all noise. The root cause was a flawed O(n²) algorithm: every entity pair sharing the same tag with |score_a - score_b| >= 3 was flagged as a contradiction.

Diagnosis path (3 bugs found):

1. Bug 1 — wrong line filter: .filter(l => l.startsWith(' Potential')) used hardcoded 2-space indent. Fixed: trim().startsWith('Potential').
2. Bug 2 — regex capturing wrong occurrence: Stats line [[!contradiction] markers: 22 and header Existing [!contradiction] tags: both matched the regex. Stats changed from tagged: → markers: to disambiguate.
3. Bug 3 — ENOBUFS: 988K output overflowed execSync pipe buffer. Stats section (at END of output) captured via tail -c 5120.

Algorithm fix — replace noisy O(n²) with targeted detection:

• Type A: Existing [!contradiction] markers in frontmatter (real, manual)
• Type B: Same raw article cited by two entities with opposite stance keywords (genuine conflict)

# OLD — 25,197 noise:
for tag, entities in tag_groups.items():
    for i, a in enumerate(entities):
        for b in entities[i+1:]:
            if abs(a['score'] - b['score']) >= 3:
                contradictions.append(...)

# NEW — targeted only:
# (a) count [!contradiction] markers in frontmatter
# (b) for entities sharing the same raw article, detect opposite stance keywords

Key lesson: When a script reports "X issues found" and X is enormous (25K), the algorithm is wrong — not the wiki. Always run the script directly first to see the raw output before blaming the data.

Anti-Patterns — wiki-lint.mjs only counts ^[...] (caret-bracket format). Obsidian display links ≠ lint citations.

• Don't append citations to frontmatter lines — the --- YAML document marker and all key-value lines between the opening and closing --- must be excluded by line-index guard.

• Don't append citations without .md suffix — lint requires the .md file extension in the citation string.

• Don't use subagent for bulk mechanical edits (200+ files with the same pattern change). Subagent exhausts at ~50 iterations. Use execute_code with direct Python instead.

• Don't trust tag counts from broken parsers. Empty tags: [] with [''].count('') can produce misleading "200 ':' tags". Verify with proper frontmatter parsing.

• Don't git add -A in a wiki — it stages everything including node_modules and _archive. Use explicit file paths.

• Don't commit without lint — always run node scripts/wiki-lint.mjs . and expect 0 errors.

• Don't use sed on macOS for replacements with complex characters — sed on macOS fails with "No such file or directory" when the replacement string contains characters that conflict with shell escaping, and the file may be partially corrupted. Always use Python execute_code for text replacements with Unicode/chinese characters.

• references/2026-05-25-wiki-audit-session.md — audit session findings: wikilink bug root cause, citation batch fix algorithm, YAML corruption patterns, real numbers

• references/wiki-quality-issues.md — 格式化工具对比、prettier 风险评估、双 frontmatter 修复、wikilink 断链诊断（bare concepts/ 名称、.md 后缀错误）

• references/duplicate-entity-cluster-diagnosis.md — 同一文章多个 entity 变体（中文/英文/v2 slug）检测、合并策略、防重检查

• references/wiki-markdown-beautifier.md — ^[raw/articles/...] 显示标记清除算法、before/after 对比、工具可用性

• references/2026-05-25-comprehensive-quality-sprint.md — 全面质量提升 Sprint 记录

• references/360-audit-findings.md — 360 audit findings: diagnostic commands, known issue patterns (WL_RE false positives, malformed updated fields, anomalous review_value, truncated index entries, missing index entries)

EXCESS INFERRED — Batch Citation Fix (Updated 2026-05-25)

Scripts (use in this order of preference):

• scripts/fix-excess-inferred-v5.mjs ← USE THIS (correct: always appends to every prose paragraph, never skips)
• scripts/fix-excess-inferred-v4.mjs ← BROKEN — DO NOT USE (skips entire file if citation already exists anywhere in body)
• scripts/fix-excess-inferred-v3.mjs ← legacy, less robust

v4 Bug Explained: v4 had if (content.includes(citation)) { skipped++; return; } — this skips the entire file if ANY paragraph already has that citation, even if 90% of other paragraphs are uncited. v5 fixes this by checking if (line.includes(citation)) continue; per-line instead of per-file.

Pre-requisite input files (v5 reads these, not raw_source_map.json):

/tmp/raw_source_map_v2.json   # slug -> raw/articles/SLUG.md filesystem map (1373 entries)
/tmp/excess_files.txt         # one "EXCESS INFERRED: entities/SLUG has N/M..." per line

Build input files:

import subprocess, os, re, json

wiki = '/Users/jinguo/wiki'
# Build raw_source_map_v2.json from filesystem
raw_files = {}
for fn in os.listdir(os.path.join(wiki, 'raw/articles')):
    if fn.endswith('.md'):
        slug = fn[:-3]
        raw_files[slug] = f'raw/articles/{fn}'
with open('/tmp/raw_source_map_v2.json', 'w') as f:
    json.dump(raw_files, f)

# Get current EXCESS INFERRED list from lint
result = subprocess.run(['node', 'scripts/wiki-lint.mjs', '.'], cwd=wiki, capture_output=True, text=True)
excess_lines = [l for l in result.stdout.split('\n') if 'EXCESS INFERRED' in l]
with open('/tmp/excess_files.txt', 'w') as f:
    for l in excess_lines: f.write(l + '\n')

v5 usage:

cd ~/wiki && node scripts/fix-excess-inferred-v5.mjs
# Output: Fixed: N entities | Total citations added: M | No source: K

# After each batch:
node scripts/wiki-lint.mjs . 2>&1 | grep -E "error\(s\)|EXCESS INFERRED" | head -5
git add entities/ && git commit -m "fix: add citations to N entities (EXCESS INFERRED batch)"

No-source entities (138 remaining as of 2026-05-25) — these cannot be auto-fixed because no raw/articles/SLUG.md exists:

• 60 RAW_FRONT: frontmatter sources: points to a slug with no corresponding file — need manual slug resolution
• 29 NO_SOURCE_FIELD: no source/sources in frontmatter — likely never had a raw saved
• 14 NEWSLETTER: source: newsletter — no specific article, newsletter-level attribution only
• 13 URL: source_url: present but no local raw file
• 9 OTHER: WeChat URLs, empty sources: [], or hybrid sources

For NO_SOURCE_FIELD and NEWSLETTER entities with high quality content: consider converting type: entity → type: concept to remove the raw-source constraint.

Broken citation cleanup (run after any batch that introduces wrong citations):

# Remove BROKEN CITATIONs introduced by bad frontmatter fallback
result = subprocess.run(['node', 'scripts/wiki-lint.mjs', '.'], cwd=wiki, capture_output=True, text=True)
output = result.stdout + result.stderr
broken_map = {}
for l in output.split('\n'):
    if 'BROKEN CITATION:' not in l: continue
    m = re.search(r'BROKEN CITATION: (entities/\S+) cites "(raw/[^"]+)"', l)
    if m:
        entity = m.group(1).replace('entities/', '')
        src = m.group(2)
        broken_map.setdefault(entity, set()).add(src)

import re, os
for entity, broken_srcs in broken_map.items():
    fp = os.path.join(wiki, 'entities', entity + '.md')
    if not os.path.exists(fp): continue
    with open(fp) as f: content = f.read()
    original = content
    for src in broken_srcs:
        content = re.sub(r'\^\[' + re.escape(src) + r'\]', '', content)
    if content != original:
        with open(fp, 'w') as f: f.write(content)

Duplicate Footnote Refs Deduplication

Problem: 1,396 entity files have the same ^[raw/articles/...] citation appearing dozens or hundreds of times in one file (one per paragraph). Example: multi-agent-trading-system-deep-thinking.md had 416 identical citations.

Detection:

import os, re
from collections import Counter

entities_dir = '/Users/jinguo/wiki/entities'
dups = []
for fname in sorted(os.listdir(entities_dir)):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, encoding='utf-8') as f:
        content = f.read()
    footnotes = re.findall(r'\^\[([^\]]+)\]', content)
    fn_counts = Counter(footnotes)
    dup = {fn: c for fn, c in fn_counts.items() if c > 1 and fn.startswith('raw/')}
    if dup:
        dups.append((fname, sum(dup.values()), dup))

print(f"Files with duplicate footnotes: {len(dups)}")
dups.sort(key=lambda x: x[1], reverse=True)
for fname, total, _ in dups[:5]:
    print(f"  {fname}: {total} duplicates")

Fix: For each file, keep only the first occurrence of each unique footnote, remove all subsequent occurrences:

import os, re

entities_dir = '/Users/jinguo/wiki/entities'
for fname in sorted(os.listdir(entities_dir)):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, encoding='utf-8') as f:
        content = f.read()
    original = content

    lines = content.split('\n')
    new_lines = []
    seen_footnotes = {}  # fn -> first line index
    for line in lines:
        fn_matches = re.findall(r'\^\[([^\]]+)\]', line)
        if fn_matches:
            for fn in fn_matches:
                if fn in seen_footnotes:
                    # Duplicate — remove just the footnote marker from this line
                    line = re.sub(r'\^\[' + re.escape(fn) + r'\]', '', line)
                    line = line.rstrip()
                    if not line.strip():
                        line = None
                        break
                else:
                    seen_footnotes[fn] = True
        if line is not None:
            new_lines.append(line)

    if ''.join(new_lines) != content.replace('\n\n\n', '\n\n'):
        content = '\n'.join(new_lines)
        content = re.sub(r'\n{3,}', '\n\n', content)
        with open(fpath, 'w', encoding='utf-8') as f:
            f.write(content)

print(f"Deduplicated footnotes in {fixed} files")

Result: 46,172 duplicate footnote instances removed across 1,401 files (commit 77e88228).

Duplicate Wikilink Deduplication

Problem: 161 entity files have the same entity/concept wikilink appearing 2+ times (same [[entity-name]] in body text multiple times). Unlike footnotes, some repetition may be legitimate (same concept discussed in different sections), so only remove 3+ occurrences.

Fix:

import os, re
from collections import Counter

entities_dir = '/Users/jinguo/wiki/entities'
fixed = 0
for fname in sorted(os.listdir(entities_dir)):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, encoding='utf-8') as f:
        content = f.read()

    wikilinks = re.findall(r'\[\[([^\]|]+?)(?:\|[^\]]+)?\]\]', content)
    entity_links = [w.strip() for w in wikilinks if not w.strip().startswith('raw/')]
    link_counts = Counter(entity_links)
    dups = {link: c for link, c in link_counts.items() if c > 2}  # only 3+

    for dup_link, count in dups.items():
        pattern = r'\[\[' + re.escape(dup_link) + r'(?:\|[^\]]+)?\]\]'
        matches = list(re.finditer(pattern, content))
        if len(matches) > 2:
            for m in reversed(matches[2:]):  # keep first 2, remove rest
                content = content[:m.start()] + '___RM___' + content[m.end():]
    if '___RM___' in content:
        content = content.replace('___RM___', '')
        content = re.sub(r'\n{3,}', '\n\n', content)
        with open(fpath, 'w', encoding='utf-8') as f:
            f.write(content)
        fixed += 1

print(f"Fixed wikilink duplicates in {fixed} files")

Result: 324 duplicate wikilink instances removed across 161 files (commit 77e88228).

NO FRONTMATTER — Missing Closing ---

Symptom: wiki-lint.mjs reports NO FRONTMATTER: entities/FILENAME even when a --- block appears to exist.

Root Cause: wiki-lint.mjs uses FM_RE = /^---\n([\s\S]*?)\n---/ — requires BOTH opening --- on its own line AND closing --- on its own line, with exactly one newline before and after each. If the closing --- is missing, or frontmatter ends directly with a YAML value without a trailing newline before the body, the regex fails to match.

Detection:

import os, re

for fn in os.listdir('entities'):
    if not fn.endswith('.md'): continue
    with open(f'entities/{fn}', encoding='utf-8') as f:
        content = f.read()
    fm_match = re.match(r'^---\n([\s\S]*?)\n---', content)
    if not fm_match:
        print(f"MISSING CLOSER: entities/{fn}")
    elif not fm_match.group(1).strip():
        print(f"EMPTY CLOSER: entities/{fn}")

Fix: Ensure proper frontmatter structure — the YAML block must have a closing --- on its own line, with at least one blank line between it and the first body heading:

---
title: ...
type: entity
sources:
  - raw/articles/...

---

# Body Heading

Specific fixes:

1. Missing closing ---: Add --- on its own line after the last YAML line
2. No blank line after ---: Insert a blank line between closing --- and the first # Heading
3. Trailing whitespace on --- line: Ensure --- is alone with no trailing spaces

Real example fixed: claude-code-large-codebase-harness-configuration.md — closing --- was absent because the sources: list ended directly above the body heading. Fixed by inserting ---\n\n between sources block and heading (commit 9a25e616).

Self-Citation Entities — Body Copies from sources:

Symptom: Entity file where sources: [raw/articles/SLUG] and body is the raw article verbatim. Every paragraph gets ^[raw/articles/SLUG.md] self-citation. Lint flags 100% uncited paragraphs (EXCESS INFERRED), and Obsidian Local Graph shows hundreds of self-links.

Root cause: LLM synthesis task was essentially copy-paste with per-paragraph citation. The file IS the source with citations pointing to itself.

Fix: Delete the self-citation markers ^[raw/articles/SLUG.md] from the body. The frontmatter sources: already provides provenance. Per-paragraph self-citations add no value when the entire body comes from that source.

# Remove all self-citation markers from an entity
import re
with open(fp) as f: content = f.read()
# Remove self-citations: ^[raw/articles/SELF_SLUG.md]
content = re.sub(r'\s+\^\[raw/articles/SELF_SLUG\.md\]', '', content)
with open(fp, 'w') as f: f.write(content)

When NOT to fix: If the entity has genuine LLM synthesis (analysis, synthesis, original insights) layered on top of source citations, keep the citations — they serve a real provenance purpose.

Real case: entities/十年老技术开发的-ai-agent-探索之路.md — 352 self-citations removed, 0 remaining. Commit 49d2a4bf.

Duplicate Entity Variants — Same Article, Multiple Slugs

Symptom: One article stored as 5 entity files with different slugs (e.g., 十年老技术开发的-ai-agent-探索之路, 十年老技术开发的-ai-agent-探索之路-v2, ai-agent-exploration-legacy-dev, ai-agent-exploration-legacy-developer, ai-agent-exploration-path-legacy-tech). Each "Related entities" section links to the other 4 variants.

Impact: Confusing cross-linking, Obsidian Local Graph shows same-article self-references across variants, index.md has 5 entries for one article.

Detection:

# Group entity files by frontmatter title or first-line heading
import os, re
groups = {}
for fn in os.listdir('entities'):
    if not fn.endswith('.md'): continue
    with open(f'entities/{fn}') as f:
        content = f.read()
    m = re.search(r'^#\s+(.+)$', content, re.MULTILINE)
    if m:
        title = m.group(1).strip()
        groups.setdefault(title, []).append(fn)

dups = {t: fs for t, fs in groups.items() if len(fs) > 1}
for t, fs in sorted(dups.items(), key=lambda x: -len(x[1])):
    print(f"{t}: {len(fs)} variants → {fs}")

Strategy: Consolidate to one canonical entity. Other variants should either be deleted (if content is redundant) or merged into the canonical with proper differentiation.

Category 1: Bare Concept/Term Links (most common)

Entity files often contain [[Agent]], [[LLM]], [[OpenClaw]], [[Claude Code]], [[Codex]], [[Y Combinator]] etc. — these lint as BROKEN because no such wiki entry exists. Convert to plain text.

Category 5: Bare concepts/ / queries/ / comparisons/ Names (RESOLVED)

Symptom: 13 broken links across 6 files — wikilinks like [[agent-memory-lifecycle-philosophies]] resolve to entities/name.md which does not exist; the actual file lives in concepts/name.md.

Fix (string replace, not regex — hyphens in names break regex char classes on macOS):

broken_map = {
    'agent-memory-lifecycle-philosophies': 'concepts/agent-memory-lifecycle-philosophies',
    'agent-memory-system-design': 'concepts/agent-memory-system-design',
    'context-management-agent-systems': 'concepts/context-management-agent-systems',
    'autonomous-agent-systems': 'concepts/autonomous-agent-systems',
    'multi-agent-systems': 'concepts/multi-agent-systems',
    'inference-optimization': 'concepts/inference-optimization',
    'agent-evaluation-benchmark-frameworks': 'concepts/agent-evaluation-benchmark-frameworks',
    'openclaw-architecture': 'concepts/openclaw-architecture',
}
for fname, replacements in source_files.items():
    content = open(f'entities/{fname}').read()
    for broken_name, prefix in broken_map.items():
        content = content.replace(f'[[{broken_name}]]', f'[[{prefix}]]')
        content = content.replace(f'[[{broken_name}|', f'[[{prefix}|')
    open(f'entities/{fname}', 'w').write(content)

Resolved: commit ef694e6a — all 13 links fixed.

Category 6: Wikilink with .md Extension

Symptom: 2 files had wikilinks with erroneous .md suffix (e.g. [[queries/llm-training-rl-research.md]]). Obsidian appends .md again → broken.

Fix: String remove .md before the closing ]] or |.

# Remove .md suffix from wikilinks
content = content.replace('[[queries/llm-training-rl-research.md]]',
                         '[[queries/llm-training-rl-research]]')

Resolved: commit ef694e6a.

Category 7: Plain-Text Bullets in ## 相关实体 Sections

Symptom: Entity files have ## 相关实体 sections where bullets are plain text (e.g. - Anthropic 联创：2028 年实现 AI 自我构建的概率超过 60%) rather than [[entities/slug|显示名]] wikilinks. These produce zero bidirectional graph connections — Obsidian shows no backlinks, no graph edges.

Detection:

import re, os
base = "/path/to/wiki/entities"
broken_per_file = {}
for fname in os.listdir(base):
    if not fname.endswith(".md"): continue
    fpath = os.path.join(base, fname)
    with open(fpath, encoding="utf-8") as f:
        lines = f.readlines()
    in_rel = False
    for i, line in enumerate(lines):
        if "## 相关实体" in line or "## Related" in line or "## 相关页面" in line:
            in_rel = True; continue
        if not in_rel: continue
        if line.startswith("## ") or line.startswith("# "): in_rel = False; continue
        stripped = line.strip()
        if not stripped or stripped.startswith(">"): continue
        if stripped.startswith("-") and "[[" not in stripped:
            broken_per_file.setdefault(fname, []).append((i+1, stripped[:100]))

Common patterns:

Pattern	Example	Fix
Recurring article title (318 occurrences, 12 texts across 42 files)	- Anthropic 联创：2028 年实现 AI 自我构建的概率超过 60%	Map to existing entity slug, replace with [[entities/slug|title]]
Empty bullet	- or -	Remove
Em-dash sub-description	- — tool注册表元数据与实际行为间的验证断层	Remove
Section divider	--- or --	Remove
Bare URL	- 控制台：https://...	Remove
Company/org bullet with description	- **Anthropic** — 治理者岗位占比最高的公司之一	Remove if no entity exists
Code field item	- `requirements`：精确关联到 REQUIREMENTS.md	Remove
RSS navigation item	- - 次日AI速递	Remove
Deleted topic map reference	- ai industry news topic map（已删除）	Remove

Fix strategy (in this order):

1. Build text→slug mapping by searching entity filenames and frontmatter titles for each plain text
2. Bulk replace the 12 high-frequency recurring texts (318 occurrences across 42 files):

   item_map = [
       ("Anthropic 联创：2028 年实现 AI 自我构建的概率超过 60%",
        "anthropic-联创2028-年实现-ai-自我构建的概率超过-60"),
       # ... all 12
   ]
   for text, slug in item_map:
       pattern = re.compile(r'^(- ' + re.escape(text) + r')$', re.M)
       replacement = f'- [[entities/{slug}|{text}]]'
       new_content = pattern.sub(replacement, content)
   

3. Bulk remove empty bullets, em-dash continuations, dividers, URLs, RSS nav items, deleted refs — one pass per file
4. Remaining orphans — find entity slug via title search, or remove if no entity exists

Real outcome: 318 high-freq text fixes + 122 removals across ~180 entity files → 0 lint errors.

Key lesson: Plain-text related-entity bullets are invisible in the graph. Well-linked body text does NOT protect against unlinked ## 相关实体 bullets — always scan that section specifically.

Common bare links that should be plain text:

• Company/org names without a wiki page: [[Y Combinator]] → Y Combinator
• Generic terms: [[Agent]] → Agent, [[LLM]] → LLM, [[RAG]] → RAG
• Product names without wiki pages: [[OpenClaw]] → OpenClaw, [[Claude Code]] → Claude Code
• Tool names: [[jq]] → jq, [[FFmpeg]] → FFmpeg, [[SQLite]] → SQLite
• Academic institutions: [[Stanford University]] → Stanford University

Python fix pattern:

from pathlib import Path

f = Path("/path/to/entity.md")
text = f.read_text()

bare_terms = ["Agent", "LLM", "OpenClaw", "Claude Code", ...]
for term in bare_terms:
    old = f"[[{term}]]"
    if old in text:
        text = text.replace(old, term)
        print(f"OK: {term}")

f.write_text(text)

Category 2: Entity References to Deleted/Non-existent Files

When an entity file references another entity that was deleted or never existed (e.g., [[entities/claude-code-开发负责人...]]), the lint reports a BROKEN LINK. Convert to plain text describing the reference, or redirect to a similar existing entity.

# Replace broken entity wikilink with plain text
text = text.replace(
    "[[entities/deleted-entity]]",
    "deleted-entity description"
)

Category 3: Footnote Refs with Wikilink Brackets

[[^4]], [[^7]] etc. appearing in text — footnote references accidentally wrapped in [[]]. Remove brackets:

text = text.replace("[[^4]]", "[^4]")

Category 4: WL_RE False Positive — Code Literals with [[...]]

Symptom: BROKEN LINK: entities/X -> [["macd", "rsi_14", ...]] — Python/JavaScript list/dict literals inside code blocks matched as wikilinks.

Root cause: WL_RE = /\[\[([^|\]#]+?)(?:\|[^\]]*?)?(?:#[^\]]*?)?\]\]/g matches [[...]] anywhere, including code.

Fix: Restructure code to avoid the [[...]] pattern:

# Bad
indicators = stock_df[["macd", "rsi_14", ...]]

# Good — use .loc or list() wrapper
indicators = list(stock_df.columns[["macd", "rsi_14", ...]])
# Or
indicators = stock_df.loc[:, ["macd", "rsi_14", ...]]

Finding All Broken Links — Full Audit Pattern

node scripts/wiki-lint.mjs . 2>&1 | grep "BROKEN LINK" | sed 's/.* -> //' | sort | uniq -c | sort -rn

This shows the broken target and count. For each unique broken target:

1. Check if the target exists as a file: ls entities/TARGET.md or ls concepts/TARGET.md
2. If exists → fix the path in the entity file
3. If NOT exists → replace [[TARGET]] with plain text

Batch Mechanical Edits — Use execute_code, Not Subagents

When fixing the same pattern across 200+ files (citations, frontmatter typos, tag normalization, broken links), use Python execute_code directly. Subagents cap at ~50 tool calls per session; Python loops handle 200+ files in one shot.

Examples where execute_code is correct:

• Tag normalization across 186 files (commit d5079a1e)
• Newsletter "newsletter" → newsletter typo across 11 files
• SHA-256 computation + frontmatter across 20 files
• Citation append across 200+ entities

When subagents ARE appropriate: Reasoning-heavy synthesis tasks (writing ## 深度分析 sections), debugging complex failures, or orchestrating multi-step workflows. Not for mechanical search-and-replace.

Rule of thumb: If the operation is "for every file matching X, replace Y with Z" — use Python. If it's "read this file, understand it, produce new content" — use a subagent.

Subagent Bulk Broken Link Fix — Iteration Limit

When delegating broken link fixes to a subagent, expect max_iterations failure around 50 tool calls. Scope the subagent to ≤3 specific files rather than "all files with broken links." For large batches, use Python execute_code directly:

# Find all files with a specific broken link pattern
from pathlib import Path
import re
wiki = Path("/path/to/wiki")
pattern = r"\[\[X]]"  # the broken link
for f in wiki.glob("entities/*.md"):
    if pattern in f.read_text():
        # fix it

macOS Sed Corruption Recovery

If sed partially corrupts a file (fails with "No such file or directory"):

git checkout -- path/to/corrupted/file.md

Then redo the fix using Python execute_code.

---

Crisis Repair Patterns (2026-06-10 Audit Session)

Git Stash Conflict Resolution at Scale

Symptom: git status shows hundreds of files with <<<<<<< Updated upstream conflicts after interrupted operations. Lint reports while scanning a simple key errors pointing to conflict markers.

Detection:

cd ~/wiki/entities
grep -l "<<<<<<< Updated upstream" *.md | wc -l
# Example output: 1005

Resolution — bulk Python script (preserve upstream version):

import os, re

entities_dir = "/Users/jinguo/wiki/entities"
files = [f for f in os.listdir(entities_dir) if f.endswith('.md')]

for fname in files:
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    
    if "<<<<<<< Updated upstream" not in content:
        continue
    
    # Resolve: keep upstream (content before =======)
    new_content = re.sub(
        r'<<<<<<< Updated upstream\n(.*?)=======\n(.*?)>>>>>>> Stashed changes',
        lambda m: m.group(1),
        content,
        flags=re.DOTALL
    )
    
    if new_content != content:
        with open(fpath, 'w', encoding='utf-8') as f:
            f.write(new_content)

Verification: After fix, grep -l "<<<<<<<" *.md should return empty.

YAML Description Field Quote Corruption

Symptom: yaml.YAMLError: while parsing a block mapping — frontmatter has description: "... without closing quote.

Root cause: LLM synthesis inserted multi-line description with opening " but no closing " before next key.

Pattern:

description: "WalkerYu 深度解读 Dan Shipper（Every 公司）AI Native 实践：...
created: 2026-06-02   # ← quote should end before this line

Detection:

import yaml, re

for fname in os.listdir('entities'):
    if not fname.endswith('.md'): continue
    with open(f'entities/{fname}') as f:
        content = f.read()
    m = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
    if not m: continue
    try:
        yaml.safe_load(m.group(1))
    except yaml.YAMLError as e:
        print(f"YAML error in {fname}: {e}")

Fix — add missing closing quote:

lines = content.split('\n')
new_lines = []
for i, line in enumerate(lines):
    new_lines.append(line)
    # Check description line without closing quote
    if re.match(r'^description:\s*"[^"]+$', line):
        if i + 1 < len(lines):
            next_line = lines[i + 1]
            # If next line is a new key, add closing quote
            if re.match(r'^[a-z_]+:', next_line):
                new_lines[-1] = line + '"'

Frontmatter Missing --- Separators

Symptom: Lint reports NO FRONTMATTER but file clearly has YAML content at top. File has keys like created:, tags: but no --- delimiters.

Pattern:

created: '2026-06-07'
review_value: 8
type: entity
tags: [aws, llm]

# Body starts here — but no --- before or after frontmatter

Fix — wrap existing YAML with ---:

for fname in files:
    with open(fname, 'r') as f:
        content = f.read()
    
    if content.startswith('---\n'):
        continue  # Already has frontmatter
    
    # Find where YAML ends (first empty line or non-key line)
    lines = content.split('\n')
    yaml_end = 0
    for i, line in enumerate(lines):
        if line.strip() == '' or ':' not in line:
            yaml_end = i
            break
    
    if yaml_end > 0:
        new_content = '---\n' + '\n'.join(lines[:yaml_end]) + '\n---\n' + '\n'.join(lines[yaml_end:])
        with open(fname, 'w') as f:
            f.write(new_content)

Bulk Broken Citation Cleanup

Symptom: Lint reports BROKEN CITATION: entities/X cites "raw/articles/Y.md" which does not exist — entity references raw file that was never ingested or was deleted.

Strategy: Remove citations pointing to non-existent raw files (keep entity content).

import os, re

wiki = "/Users/jinguo/wiki"
raw_dir = os.path.join(wiki, "raw/articles")
entities_dir = os.path.join(wiki, "entities")

existing_raw = {f[:-3] for f in os.listdir(raw_dir) if f.endswith('.md')}

for fname in os.listdir(entities_dir):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    
    # Find all citations
    citations = re.findall(r'\^\[raw/articles/([^\]]+)\]', content)
    
    for cite in citations:
        raw_slug = cite.replace('.md', '')
        if raw_slug not in existing_raw:
            # Remove broken citation
            content = content.replace(f'^[raw/articles/{cite}]', '')
    
    with open(fpath, 'w', encoding='utf-8') as f:
        f.write(content)

Bulk Broken Wikilink → Plain Text

Symptom: BROKEN LINK: entities/X -> [[entities/Y]] — target entity doesn't exist.

Strategy: Convert to plain text (keep display text if any).

# Build existence sets
existing_entities = {f[:-3] for f in os.listdir(entities_dir) if f.endswith('.md')}

for fname in os.listdir(entities_dir):
    fpath = os.path.join(entities_dir, fname)
    with open(fpath, 'r', encoding='utf-8') as f:
        content = f.read()
    
    # Find all entity wikilinks
    wikilinks = re.findall(r'\[\[entities/([^\]|]+)(?:\|([^\]]+))?\]\]', content)
    
    for slug, display in wikilinks:
        if slug not in existing_entities:
            # Convert to plain text
            old = f'[[entities/{slug}|{display}]]' if display else f'[[entities/{slug}]]'
            new = display if display else slug.replace('-', ' ')
            content = content.replace(old, new)
    
    with open(fpath, 'w', encoding='utf-8') as f:
        f.write(content)

Rule of thumb: Bulk operations >50 files → Python execute_code. Subagents cap at ~50 tool calls.

Support Scripts

These scripts live under scripts/ and are referenced here for discoverability:

• scripts/wiki-lint.mjs — primary validation; always run after changes
• scripts/fix-excess-inferred-v5.mjs ← current working script; appends ^[raw/articles/SLUG.md] citations to all prose paragraphs. Requires /tmp/raw_source_map_v2.json and /tmp/excess_files.txt (see excess-inferred-playbook reference).
• scripts/fix-excess-inferred-v3.mjs ← legacy; deprecated, prefer v5
• scripts/fix-excess-inferred-v4.mjs ← broken; do not use (skips entire file if citation exists anywhere in body)

Related Skills & References

This umbrella absorbed several narrow one-shot skills during the 2026-06-03 consolidation. Their content is preserved here as references:

• references/wiki-log-md-safety.md — merged from wiki-log-append + wiki-log-maintenance: 5 patch patterns that corrupt log.md, append-only invariants, cron-mode workaround for execute_code BLOCKED
• references/wiki-cron-report-and-gitignore.md — merged from wiki-cron-status-report + wiki-gitignore-assessment: cron status report email pipeline + .gitignore assessment framework (which dirs to ignore, Obsidian mixed content rules)

Content Expansion — Concept Pages

Expand thin concept pages (4–7KB, low entity-link density) into rich 10KB+ synthesis artifacts with verified cross-references. Concepts are the skeleton of the wiki — they define terminology and link multiple entities together.

Concept vs Entity vs Query — The Three Layers

Layer	Purpose	Source	Quality bar
entities/	Source bridges — raw article summaries with citations	Ingested from articles	原文存档 + 深度分析
concepts/	Terminology/framework definitions — synthesize multiple entities	Original synthesis	10KB+, 3+ entity links, sections
queries/	Research question answers — what does this wiki know about X?	Original synthesis	Real question + evidence

raw/ → entities/ → concepts/ → queries/
         (素材)     (骨架)       (入口)

Golden rule: concept 是 wiki 区别于一堆 bookmarks 的核心价值 — 汇聚散落在多个 entity 页里的同一概念。

Content Expansion — Adding Deep Analysis & Practical Insights to Entities

When expanding existing entity pages with ## 深度分析 and ## 实践启示 sections, follow this pattern:

Insert location: BEFORE the → [[raw/articles/...|原文存档]] line (the last line before end of body).

Structure (5 subsections each):

## 深度分析
### 1. [Analytical theme with specific technical insight]
[Paragraph with ^[raw/articles/SLUG.md:LINE_RANGE] citation] ^[raw/articles/alibaba-agent-observability-audit-loongsuite-pilot-coding-agent-blackbox-transparent.md:153-155]

### 2. [Second analytical theme]
...

## 实践启示
### 1. [Actionable recommendation]
...

### 2. [Second actionable item]
...

Line citation rules:

• Cite ^[raw/articles/SLUG.md:LINE_RANGE] using the raw article line numbers (not the entity file)
• Each analytical point should reference 1-3 specific line ranges from the raw source