wiki-audit-and-repair

name	wiki-audit-and-repair
description	Systematically audit and repair an Obsidian wiki/index: fix frontmatter drift, missing index entries, broken wikilinks, log rotation, and SCHEMA.md drift.

Wiki Audit & Repair

• references/index-dedup-pitfall.md — index.md + index-sources.md overlap causing mass DUPLICATE errors (2026-06-27)
• references/wiki-regex-pitfalls.md — regex pitfalls (2026-05-23)
• references/batch-ingest-false-rejection-postmortem.md — evaluate batch ingest false rejections via git post-mortem (2026-07-04)

Phase 0: Trust Nothing — Verify REVIEW.md Against Disk

CRITICAL PITFALL: When REVIEW.md (or any external status doc) claims cleanup phases are "已完成 ✅", do NOT trust it. The index.md might have been rebuilt from a disk scan that still contains all pre-cleanup files. Always verify:

1. Compare REVIEW.md's claimed entity count against actual ls entities/*.md | wc -l
2. Run lint — if lint shows pre-cleanup tracked page count, the claimed deletions never happened
3. Run the quality classification script to confirm actual tier distribution

Critical: execute node scripts/wiki-lint.mjs FIRST — lint output is the ground truth for all metrics. Do not trust REVIEW.md, memory, or prior session state.

Comprehensive Health Check Workflow

Trigger phrases: "360度体检", "全面体检", "wiki health check", "wiki audit"

When asked to do a "full health check" or "全面体检", follow this systematic approach:

1. Run node scripts/wiki-lint.mjs — this is the single source of truth for error count and page counts
2. Execute the diagnostic commands in references/360-audit-findings.md
3. Build a before/after metric table
4. Fix in priority order: lint errors → index drift → broken wikilinks → frontmatter → expansion gaps
5. Always run sed -i '' 's/^|- /- /' <file> after any patch to frontmatter or index.mdcute_code read_file() cache pollution.** Within the same session, calling read_file() on the same file multiple times causes Hermes to return cached "File unchanged" content with line-number prefixes (e.g., 1|CONTENT). This corrupts subsequent write_file() operations if the cached content is used as the base. Safe pattern: for file content verification inside execute_code, always use open() directly — never rely on a prior read_file() call's return value as the source of truth for the same file in the same session.

Typical scenario: REVIEW.md says "30 empty + 269 stubs deleted, 890 entities remaining" but lint shows 1204 entities and 2411 tracked pages. The deletions were documented but never committed to disk. The index was rebuilt from the actual files on disk, restoring all entries.

Fix: Re-run the quality classification fresh, present actual findings to user, and execute deletions in stages (smallest/emptiest first, then stubs with backlink cleanup).

Phase 1: Inventory & Baseline

1. Count all files in each directory (entities, concepts, comparisons, queries, raw/articles)
2. Run node scripts/wiki-lint.mjs <wiki_root> to get authoritative error/warning counts
3. Check for hidden files in raw/articles/ (e.g., .md duplicate)
4. Check for .md suffix entries in index.md
5. Check for leading spaces in index entries
6. Check for merged lines (source + query entries on same line)

Phase 2: Fix Index Issues (in order)

1. Header count — run lint → copy tracked count → update header → re-run lint
2. Missing query entries — scan queries/ directory → append to index.md Queries section
3. Missing source entries — scan entity files for source_url → append to index.md Sources section
4. Merged lines — split concatenated entries onto separate lines
5. .md suffix entries — strip .md from index paths
6. Leading spaces — strip whitespace from entry lines
7. GHOST entries — remove entries pointing to non-existent files
8. BROKEN LINKS — find entity files with broken links → update to correct slug
9. Hidden .md duplicate — if found, compare URL against existing articles → remove or rename

Phase 3: Verify

1. Run lint again — target 0 errors
2. If errors remain, repeat Phase 2 for remaining issues
3. Commit changes with descriptive message

Phase 4: Report

Provide a summary table:

• Before: X errors, Y warnings
• After: 0 errors, 0 warnings
• Files fixed: list of issues resolved
• Remaining: any known debt or pending items

Step 1: Inventory files

import os, re
VAULT = os.path.expanduser("<wiki_root>")

def count_files(subdir):
    path = os.path.join(VAULT, subdir)
    if not os.path.exists(path):
        return 0, set()
    files = set(f.replace(".md", "") for f in os.listdir(path) if f.endswith(".md"))
    return len(files), files

raw_n, raw_files = count_files("raw/articles")
ent_n, entity_files = count_files("entities")
con_n, concept_files = count_files("concepts")
actual_total = raw_n + ent_n + con_n

Step 2: Audit index.md

Read the full index, extract all wikilinks, and count:

with open(os.path.join(VAULT, "index.md")) as f:
    content = f.read()

all_links = re.findall(r'\[\[([^\]]+)\]\]', content)
indexed_raw = set()
indexed_ent = set()
indexed_con = set()

from collections import defaultdict
seen = defaultdict(list)

for link in all_links:
    link = link.split("|")[0]
    if link.startswith("entities/"):
        indexed_ent.add(link.split("/", 1)[1])
        seen[link].append(1)
    elif link.startswith("raw/articles/"):
        indexed_raw.add(link.split("/", 2)[2])
        seen[link].append(3)
    elif link.startswith("concepts/"):
        indexed_con.add(link.split("/", 1)[1])
        seen[link].append(2)

# Check header count
m = re.search(r'Total pages: (\d+)', content)

Step 3b: Fix GHOST raw article entries (v1→v2 migration)

When lint reports GHOST: index entry [[raw/articles/ghost-name]] has no file on disk, the typical pattern is:

1. The raw article was re-ingested as ghost-name-v2.md (v2 version exists on disk)
2. Entity files still reference raw/articles/ghost-name in their sources: field
3. Index.md still has stale entries for ghost-name

Fix workflow:

import os

VAULT = os.path.expanduser("~/wiki")

# Step 1: Find ghosts — raw article slugs in index that have no file on disk
with open(os.path.join(VAULT, "index.md")) as f:
    lines = f.read().split('\n')

raw_files = set(f.replace(".md", "") for f in os.listdir(os.path.join(VAULT, "raw/articles")) if f.endswith(".md"))
ghost_raw = set()
for line in lines:
    for m in re.finditer(r'\[\[raw/articles/([^\]]+)\]\]', line):
        slug = m.group(1).split("|")[0]
        if slug not in raw_files and slug + ".md" not in raw_files:
            ghost_raw.add(slug)

# Step 2: For each ghost, check if v2 exists on disk
ghost_to_v2 = {}
no_v2 = {}
for ghost in ghost_raw:
    v2 = ghost + "-v2"
    if v2 in raw_files or os.path.exists(os.path.join(VAULT, "raw/articles", v2 + ".md")):
        ghost_to_v2[ghost] = v2
    else:
        no_v2[ghost] = True

print(f"Ghosts with v2: {len(ghost_to_v2)}, without v2: {len(no_v2)}")

Step 3: Fix entity files — update sources field from ghost → v2

# CRITICAL: Beware the double-v2 bug!
# If entity filename is already "xxx-v2.md" and you replace "xxx" → "xxx-v2",
# you get "xxx-v2-v2" which is wrong.
# Only replace "raw/articles/{ghost}" not the bare ghost slug in filenames.

for fname in os.listdir(os.path.join(VAULT, "entities")):
    fpath = os.path.join(VAULT, "entities", fname)
    with open(fpath) as f:
        content = f.read()
    
    changed = False
    for ghost, v2 in ghost_to_v2.items():
        old = f"raw/articles/{ghost}"
        if old in content:
            content = content.replace(old, f"raw/articles/{v2}")
            changed = True
    
    # For ghosts without v2, just remove the wikilink reference entirely
    for ghost in no_v2:
        old = f"raw/articles/{ghost}"
        if old in content:
            lines = content.split('\n')
            content = '\n'.join(l for l in lines if old not in l)
            changed = True
    
    if changed:
        with open(fpath, 'w') as f:
            f.write(content)

Step 4: Remove ghost entries from index.md

# Remove index entries pointing to ghost raw articles
new_lines = []
for line in lines:
    skip = False
    for ghost in ghost_raw:
        if f"raw/articles/{ghost}" in line:
            skip = True
            break
    if not skip:
        new_lines.append(line)

with open(os.path.join(VAULT, "index.md"), 'w') as f:
    f.write('\n'.join(new_lines))

Step 5: Fix duplicate index entries

from collections import defaultdict
seen = defaultdict(list)
for i, line in enumerate(lines):
    for m in re.finditer(r'\[\[([^\]]+)\]\]', line):
        slug = m.group(1).split("|")[0]
        seen[slug].append(i)

dup_slugs = [slug for slug, positions in seen.items() if len(positions) > 1]
for slug in dup_slugs:
    positions = seen[slug]
    # Keep first (positions[0]), remove rest in reverse order
    for pos in reversed(positions[1:]):
        print(f"Removing duplicate at line {pos+1}: {lines[pos][:60]}")
        lines.pop(pos)

Pattern 11: Case-sensitivity ghost slug — ghost + correct entry coexist (2026-05-21)

Root cause: A slug has a case mismatch between index.md entry (wrong case) and the actual filename (correct case). Both the ghost entry AND the correct entry may coexist in index.md, and the entity file's sources: field also carries the wrong-case slug.

Example: kimi-attention-residuals-prenorm-dilution-block-attnres (ghost, wrong) vs kimi-attention-residuals-preNorm-dilution-block-attnres (correct, file on disk).

Detection:

import os, re
from pathlib import Path

wiki = Path("/Users/jinguo/wiki")
# Find entity files whose stem differs only by case from an index entry
entity_files = {f.stem: f for f in (wiki/"entities").glob("*.md")}

with open(wiki/"index.md") as f:
    index = f.read()

for line in index.split('\n'):
    for m in re.finditer(r'\[\[entities/([^]|\]]+)', line):
        slug = m.group(1)
        slug_lower = slug.lower()
        matching = [k for k in entity_files if k.lower() == slug_lower]
        for k in matching:
            if k != slug:  # case mismatch
                print(f"  CASE MISMATCH: index=[[{slug}]] disk=[[{k}]]")

Fix workflow (3-step, must do ALL):

1. Fix entity file sources: Update raw/articles/{wrong-slug} → raw/articles/{correct-slug} in entity frontmatter/sources field
2. Fix index.md ghost entry: Replace [[entities/{wrong-slug}]] → [[entities/{correct-slug}|{title}]] (use the existing correct entry's title)
3. Check for duplicate: After replacement, verify only ONE entry for the correct slug remains in index.md

Why step 1 is critical: If you only fix the index entry, the entity file still references the ghost raw article and lint will report BROKEN LINK from entity → ghost raw.

Duplicate detection pattern: When adding 22 MISSING entities, if any share the same slug as an existing entry (e.g., preNorm already existed as correct entry and prenorm was ghost), the insertion creates a duplicate. Always check content.count(slug) before inserting.

Known ghost → v2 migration patterns (2026-05-14)

These raw articles were re-ingested as v2; update entity sources accordingly:

• normalizing-trajectory-models → normalizing-trajectory-models-v2
• openclaw-完全指南...32w字 → ...-v2
• boris-cherny-新访谈... → ...-v2
• 你不知道的-agent原理架构与工程实践 → ...-v2
• 天猫新品营销技术团队ai编码实战指南上/下 → ...-v2
• ai-tool-poisoning-exposes-... → ...-v2
• deepseek视觉原语论文... → ...-v2
• subagents-详解claude-code-... → ...-v2
• ai-coding-入门指南... → ...-v2

Ghosts with NO v2 (remove wikilink from entity, don't map):

• karpathy-vibe-coding-agentic-engineering-v2 / v3 (no v2, no file)
• untitled (minimal content — just remove link)

Step 6: Fix index issues

Fix header count mismatch: Run the linter first to get the authoritative count, then either update directly or rebuild from disk:

# 获取 linter 的权威计数
actual=$(node scripts/wiki-lint.mjs <wiki_root> 2>&1 | grep -oP '\d+(?= tracked page)')
# 用该数值直接替换 header
sed -i '' "s/Total pages: [0-9]*/Total pages: $actual/" index.md

Fix INDEX DRIFT: The header may be off by 1+ due to concurrent writes or missed updates. Always trust lint's tracked page(s) count. If the discrepancy is >0, run lint → copy count → re-run lint to confirm.

Fix GHOST entries: When lint reports GHOST: index entry [[entities/slug]] has no file on disk, the entry in index.md points to a file that no longer exists. Remove the entry from index.md. If the entity page still exists with a different name, update the link to the correct slug.

Fix BROKEN LINKS: When lint reports BROKEN LINK: entities/foo -> [[entities/bar]], find the entity file containing the broken link and update it to point to the correct slug.

Add missing query entries: Query files in queries/ directory may not have index entries. Scan for all .md files in queries/, then append entries to the Queries section of index.md.

Add missing source entries: Entity pages with source_url fields may not have corresponding [[raw/articles/...]] entries in index.md. Scan entity files for source_url, extract the raw article slug, and append missing source entries to the Sources section.

Fix merged lines: When entries are concatenated on the same line (e.g., source entries followed by query entries without newline separator), split them:

content = re.sub(r'(\]\])\s*- \[\[(entities|raw|concepts|comparisons|queries)/', r'\1\n- [[\2/', content)

Fix entries with .md suffix: Index entries must NOT include .md suffix. The lint strips .md from filenames to build its ACTIVE dict. Entries like [[raw/articles/foo.md]] will never match key raw/articles/foo. Fix:

content = re.sub(
    r'\[\[(entities|raw|concepts|comparisons|queries)/([^)|\]]+)\.md(\|[^]]*)?\]\]',
    lambda m: f'[[{m.group(1)}/{m.group(2)}{m.group(3) or ""}]]',
    content
)

Fix leading spaces: Entries with leading whitespace (e.g., - [[...]]) are invisible to lint regex ^- \[\[. Fix:

content = re.sub(r'^(\s+)- \[\[', '- [[', content, flags=re.MULTILINE)

⚠️ Lint regex requires NO leading spaces: The lint script parses index entries with ^- \[\[. Entries with ANY leading whitespace (e.g., - [[entities/foo]]) are silently ignored — they pass the duplicate check but fail the "MISSING from index" check. This creates phantom errors that seem impossible (the entry IS in the file, but lint says it's missing).

Root cause (confirmed 2026-05-20): When inserting new entity entries into index.md programmatically (e.g., via fix_8_entities.py script that appends new entries), the insertion used 2-space indentation ( - [[entities/X]]) matching a section header pattern. But lint's regex ^- \[\[ requires zero leading spaces. Result: 8 new entities appeared in index.md (visible to the human eye) but lint reported all 8 as "MISSING from index".

Detection: Run lint — if it reports N "MISSING from index" but the entries ARE visibly in index.md, check for leading spaces:

grep -n "^  - \[" index.md  # Find 2-space indented entries

Fix: Normalize all leading spaces:

content = re.sub(r'^(\s+)- \[\[', '- [[', content, flags=re.MULTILINE)

Verification: After the fix, lint should report 0 errors. Do a git diff index.md to confirm only whitespace changed.

Fix hidden .md duplicate files: A hidden file named .md in raw/articles/ can be a duplicate raw article. The lint counts it because .md.endswith('.md') is True, causing INDEX DRIFT. Fix:

# Check for hidden files in raw/articles/
import os
for f in os.listdir(raw_articles_dir):
    if f == '.md':
        # Compare URL against existing articles
        # If duplicate, remove it; if unique, rename to proper slug

Severe corruption — rebuild index.md from disk: When index.md has merged-line corruption (patch tool corrupting ]] endings causes entries to merge into giant single lines >100KB), the file becomes unreadable and cannot be patched safely. Full rebuild required:

import os, re, datetime

VAULT = os.path.expanduser("~/wiki")

sections = {
    "Sources": ("raw/articles", r'\[\[raw/articles/([^|\]]+)'),
    "Entities": ("entities", r'\[\[entities/([^|\]]+)'),
    "Concepts": ("concepts", r'\[\[concepts/([^|\]]+)'),
    "Comparisons": ("comparisons", r'\[\[comparisons/([^|\]]+)'),
    "Queries": ("queries", r'\[\[queries/([^|\]]+)'),
}

lines = [
    "# Index\n",
    f"> Last rebuilt: {datetime.datetime.now().strftime('%Y-%m-%d %H:%M')} (disk scan)\n",
    "> Read this first to find relevant pages for any query.\n",
    "> Agent pipeline: [[WORKFLOW|WORKFLOW.md]] — Triage → Score → Ingest → Index\n",
    f"\nTotal pages: {{total}}\n",
    "\n---\n",
]

total = 0
for section_name, (subdir, _) in sections.items():
    lines.append(f"\n## {section_name}\n")
    dir_path = os.path.join(VAULT, subdir)
    if os.path.exists(dir_path):
        files = sorted(f for f in os.listdir(dir_path) if f.endswith(".md"))
        total += len(files)
        for fname in files:
            slug = fname[:-3]
            title = slug
            try:
                with open(os.path.join(dir_path, fname)) as fh:
                    content = fh.read()
                m = re.search(r'^title:\s*["\']?([^"\'\n]+)["\']?\s*$', content, re.MULTILINE)
                if m:
                    title = m.group(1).strip()
            except:
                pass
            lines.append(f"- [[{subdir}/{slug}|{title}]]\n")

lines = [l.replace("{total}", str(total)) for l in lines]

with open(os.path.join(VAULT, "index.md"), "w") as f:
    f.writelines(lines)

print(f"Rebuilt index.md: {total} pages")

验证重建结果： lint 应报告 0 errors，且 Total pages header 与 tracked count 完全一致。

关键原则：永远信任 linter 的 tracked page(s) 数值，而不是从前值递增。

• 当你打开文件时看到的 Total pages 可能已经被其他 agent 污染（并发写入累积的系统性偏移）
• 不要尝试 "当前值 + 2" 或 "恢复基线再递加"——这会导致漂移无限积累
• 阶段性运行 lint 修复是最好的重置方法

并发安全提示： 多个 agent 并发操作 wiki 时，patch 和 sed 都可能因竞争条件写入错误的值。修复完成后应再跑一次 lint 确认。

Fix duplicate entries: Identify duplicates from seen dict, remove the extra line with patch.

Add missing entries: For each missing file, read its title from frontmatter and construct a summary from the body (first 120 chars). Insert into the appropriate section with patch.

Fix broken wikilinks: Scan for [[entities/something]] where the file is actually entities/something-else.md. Fix the index entry to point to the actual filename.

PITFALL: patch() with replace_all=True will match ALL occurrences including unintended ones (e.g., a raw article link appearing in both Entities and Sources sections). Use targeted unique strings or insert after a specific preceding line instead.

Pattern 4: sha256 block trapped outside frontmatter (critical — causes MISSING sha256 after bulk wikilink removal)

Root cause (confirmed 2026-05-20): When a bulk wikilink cleanup removes lines containing [[concepts/X]] from entity files, if the entity file structure is:

---
title: ...
---
# Entity Title
^[raw/articles/source.md]

sha256: abc123...
---
## 深度分析
[[concepts/X]]  ← this line gets deleted by cleanup

The sha256: line between the body and the --- separator gets moved into the frontmatter by the cleanup script's line-removal logic. Lint then reports MISSING sha256 because it looks for sha256: INSIDE the frontmatter block.

Detection:

import re
from pathlib import Path

wiki = Path("/Users/jinguo/wiki")
outside = []
for f in wiki.rglob("*.md"):
    if 'raw/articles' in str(f):  # only entities/...
        continue
    content = f.read_text(errors='replace')
    if content.count('---') < 2:
        continue
    parts = content.split('---', 2)
    if len(parts) < 3:
        continue
    body = parts[2]
    body_sha = re.search(r'^sha256:\s*([a-f0-9]{64})', body, re.MULTILINE)
    if body_sha:
        outside.append((f.name, body_sha.group(1)[:16]))

print(f"sha256 outside frontmatter: {len(outside)} files")

Fix — move sha256 back inside frontmatter:

for f in wiki.rglob("*.md"):
    content = f.read_text(errors='replace')
    if content.count('---') < 2:
        continue
    parts = content.split('---', 2)
    if len(parts) < 3:
        continue
    frontmatter = parts[1]
    body = parts[2]
    m = re.search(r'^sha256:\s*([a-f0-9]{64})', body, re.MULTILINE)
    if not m:
        continue
    sha = m.group(1)
    body = re.sub(r'^sha256:\s*[a-f0-9]{64}\s*\n', '', body, flags=re.MULTILINE)
    new_fm = frontmatter.rstrip() + '\nsha256: ' + sha + '\n'
    f.write_text(f"---\n{new_fm}---\n{body}")

Prevention: Never remove entire lines in bulk cleanup — replace [[deleted/slug]] with empty string '' rather than removing the line. This preserves structural elements in their original positions.

Pattern 5: Subagent bulk wikilink cleanup destroys 3000+ files

Root cause (2026-05-20): A subagent running wikilink cleanup was given a redirect map for 19 deleted concepts. It applied re.sub(pattern, '', content) to ALL .md files across the entire wiki including .venv/, node_modules/, .obsidian/plugins/, and skills/node_modules/. Every line containing a deleted concept was replaced with a blank line. 3119 files modified.

Detection: Run git status --short | wc -l immediately after a bulk cleanup subagent completes. Normal: 5-50 files. Catastrophic: >500.

Recovery: git checkout HEAD -- . restores all files before the subagent ran. Then re-run cleanup surgically.

Safe subagent cleanup pattern:

# SAFE: Replace broken wikilink with correct target
content = re.sub(r'\[\[deleted/slug\]\]', '[[entities/live-target]]', content)

# DANGEROUS: Remove wikilink entirely (leaves blank lines, corrupts structure)
content = re.sub(r'\[\[deleted/slug\]\]', '', content)

# DANGEROUS: Remove entire line (destroys surrounding context)
lines = [l for l in content.split('\n') if 'deleted/slug' not in l]

Scope subagents to minimum directories — never give a subagent a task that removes content from "all .md files in the wiki". Scope to only the affected directories: entities/, queries/, concepts/, index.md.

Pattern 6: Case-sensitivity wikilink drift after bulk file operations

Root cause (2026-05-20): Bulk wikilink cleanup introduced case mismatches between wikilink targets and actual filenames (e.g., [[entities/kimi-attention-residuals-prenorm-...]] vs actual file entities/kimi-attention-residuals-preNorm-...).

Detection:

import os, re
from pathlib import Path

wiki = Path("/Users/jinguo/wiki")
entity_files = {f.stem.lower(): f.stem for f in (wiki/"entities").glob("*.md")}

for f in wiki.rglob("*.md"):
    if any(x in str(f) for x in ['.git', 'node_modules', '.venv']):
        continue
    content = f.read_text(errors='replace')
    for link in re.findall(r'\[\[entities/([^]|]+)', content):
        link_lower = link.lower()
        if link_lower in entity_files and entity_files[link_lower] != link:
            print(f"  {f.name}: [[{link}]] → [[{entity_files[link_lower]}]]")

Pattern 7: macOS is case-insensitive — entity path confusion

Confirmed (2026-05-20): macOS APFS/HFS+ is case-insensitive by default. entities/KIMI.md and entities/kimi.md resolve to the same file on disk. When comparing "duplicate" files found via glob(), use os.path.samefile() to detect true duplicates vs. case-aliases.

import os
from pathlib import Path

f1 = Path("entities/KIMI.md")
f2 = Path("entities/kimi.md")
print(f"Same file: {os.path.samefile(f1, f2)}")  # True on macOS

Pattern 8: concepts/ bulk deletion creates 80+ broken links cascade

Workflow for bulk concept deletion (2026-05-20):

1. Audit first — find ALL files referencing each concept BEFORE deletion
2. Build redirect map — per-concept: redirect target or remove entirely
3. Fix references first — update all referencing files before deleting
4. Delete in small batches — 5 files at a time, run lint, fix broken links, repeat
5. Never batch-delete + batch-fix in one subagent pass

Classification (2026-05-20):

• STUB (<400 bytes, 0 wikilinks, no content) → DELETE immediately
• AI_SUMMARY (3-7KB, no original framework, just summarizes existing entities) → CHECK if referenced, DELETE if orphan or redirect if referenced
• REAL_CONTENT (has original framework definition, citations) → KEEP

# AI_SUMMARY detection script
import re
from pathlib import Path

concepts_dir = Path("/Users/jinguo/wiki/concepts")
for f in sorted(concepts_dir.glob("*.md")):
    content = f.read_text(errors='replace')
    body = content.split('---', 2)[-1] if '---' in content else content
    wikilinks = re.findall(r'\[\[[^]]+\]\]', body)
    cites = re.findall(r'\[\^[^\]]+\]', body)
    paras = [p for p in body.split('\n') if len(p) > 100 and not p.startswith('#') and not p.startswith('-')]
    print(f"{len(wikilinks):3d} links | {len(cites):3d} cites | {len(paras):3d} paras | {f.stat().st_size:6d}b | {f.name}")

Pattern 9: "Truncated slug entities" — problem was exaggerated

Finding (2026-05-20): Systematic scan found 0 entity files with slug ≥200 chars. The previously-reported "12 truncated slug duplicates" were actually different articles sharing a long prefix (e.g., harness-engineering prefix of 4 different articles). No OS-level filename truncation exists in the current corpus.

Real deduplication needs:

1. Tether file variants — multiple slugs for same article from 微信 title truncation. Deduplicate by keeping the largest file.
2. True content duplicates — compare sha256 of body content:

import hashlib
from pathlib import Path
from collections import defaultdict

entities_dir = Path("/Users/jinguo/wiki/entities")
body_hashes = defaultdict(list)
for f in entities_dir.glob("*.md"):
    content = f.read_text(errors='replace')
    if '---' not in content:
        continue
    body = content.split('---', 2)[-1]
    body_hashes[hashlib.sha256(body.encode()).hexdigest()].append(f.name)

duplicates = {h: files for h, files in body_hashes.items() if len(files) > 1}

Pattern 10: Adding MISSING entity/raw entries — systematic insertion (2026-05-21)

When lint reports MISSING from index: entities/X but the file exists on disk, the entity is simply not referenced in index.md. This commonly happens after new ingestions. The fix: insert entries at the correct alphabetical position.

Step 1: Gather missing entries with titles

import os
wiki = "/path/to/wiki"
missing_entities = ["slug1", "slug2", ...]

entities_path = f"{wiki}/entities"
entries = []
for slug in missing_entities:
    ep = f"{entities_path}/{slug}.md"
    title = slug
    with open(ep, encoding='utf-8', errors='ignore') as f:
        for line in f:
            if line.startswith('title:'):
                title = line.split(':', 1)[1].strip().strip('"')
                break
    entries.append((slug, title))

Step 2: Check for existing duplicates BEFORE inserting

with open(f"{wiki}/index.md") as f:
    content = f.read()

for slug, title in entries:
    # Entity might already exist in index under a DIFFERENT section
    # (e.g., raw link in Entities section, or entity already in Queries section)
    if f'[[entities/{slug}|' in content or f'[[entities/{slug}]]' in content:
        print(f"ALREADY INDEXED (skip): {slug}")
        entries.remove((slug, title))
    if f'[[raw/articles/{slug}|' in content:
        print(f"ALREADY INDEXED as raw (skip): {slug}")
        entries.remove((slug, title))

Step 3: Sort and find insertion positions (alphabetical)

entries.sort(key=lambda x: x[0])  # sort by slug

# Find Entities section boundaries (1-indexed line numbers)
lines = content.split('\n')
entity_start = entity_end = None
for i, line in enumerate(lines):
    if line.strip() == '## Entities':
        entity_start = i + 1
    elif entity_start and line.startswith('## '):
        entity_end = i
        break

# For each entry, find where it should be inserted
insertions = []
for slug, title in entries:
    pos = entity_end  # default: append at end
    for i in range(entity_start, entity_end):
        # Extract slug from existing entry
        import re
        m = re.match(r'^(- )\[\[entities/([^\]|]+)(?:\|[^\]]+)?\]\]( — .*)?$', lines[i])
        if m and m.group(2) > slug:
            pos = i
            break
    insertions.append((slug, title, pos))

Step 4: Insert in REVERSE order (critical!) Inserting at the same position shifts all subsequent insertions. Always insert from last to first:

# Group by insertion line
from collections import defaultdict
by_line = defaultdict(list)
for slug, title, pos in insertions:
    by_line[pos].append((slug, title))

# Process in descending line number order
for lineno in sorted(by_line.keys(), reverse=True):
    for slug, title in by_line[lineno]:
        link = f"- [[entities/{slug}|{title}]]\n"
        lines.insert(lineno - 1, link)  # lineno is 1-indexed

with open(f"{wiki}/index.md", 'w') as f:
    f.write('\n'.join(lines))

Step 5: Handle same-section bleed — entities appearing in Sources section Sometimes an entity's raw link appears in the Entities section (wrong section). After inserting the correct entity entry, delete the wrong-section duplicate:

# Check for same slug appearing twice in different sections
import re
slug_counts = {}
for i, line in enumerate(lines):
    for m in re.finditer(r'\[\[entities/([^)|\]]+)', line):
        slug = m.group(1)
        slug_counts.setdefault(slug, []).append(i + 1)

for slug, positions in slug_counts.items():
    if len(positions) > 1:
        print(f"DUPLICATE: {slug} at lines {positions}")
        # Keep the first (correct section), remove the rest

Pattern 13: c=0 false alarm — eval script reads wrong field names

Root cause (confirmed 2026-05-20): The quality classification eval script reads value and confidence from frontmatter, but actual wiki frontmatter fields are review_value and review_confidence. Every entity reports v×c=0 regardless of actual content quality.

eval script fields vs actual fields:

eval reads	actual wiki field
value	review_value
confidence	review_confidence

Detection: When node -e "const v = parseInt(get('value')...)" returns 0 for every entity, the frontmatter doesn't have value: field — check for review_value: instead.

Fix: Rewrite eval script to read review_value and review_confidence:

// WRONG — reads non-existent fields
const v = parseInt(get('value') || '0');
const c = parseInt(get('confidence') || '0');

// CORRECT
const v = parseInt(get('review_value') || get('value') || '0');
const c = parseInt(get('review_confidence') || get('confidence') || '0');

Impact: An entire batch of 39 "c=0" entities appeared to be unscored, but all actually have review_value=5-9. They needed scoring verification, not deletion.

Pattern 15: Phantom BROKEN LINK — ghost target name appears in lint output but grep finds nothing

Scenario (confirmed 2026-05-21): Lint reports BROKEN LINK: queries/ai-skill-design-topic-map -> [[concepts/mcp-skills]], but grep -rn "concepts/mcp-skills" queries/ai-skill-design-topic-map.md returns nothing.

Root cause: Lint resolves entities/skillmd-zuo-liao-yi-ge-jian-li-sheng-cheng-qi (broken, wrong slug) and concepts/mcp-skills (phantom) as two separate broken link candidates, but the actual line 84 is an entities/...-mcp-skills wikilink that lint mis-parses as concepts/mcp-skills. Fixing the entity slug makes the phantom disappear — no separate concepts/mcp-skills entry ever existed.

Detection when phantom suspected:

1. grep -rn "<phantom-target>" <file> returns empty — the broken link doesn't appear in the file at all
2. A nearby correct wikilink exists with a similar slug pattern that could be mis-parsed
3. Fixing the nearby correct wikilink resolves BOTH broken links

Fix: Read the file around the reported line number and look at the actual wikilinks nearby — one wrong slug is causing lint to hallucinate a second phantom error.

Pattern 14: Frontmatter field-setting corruption — cascading merge when patching multiple fields in Node.js

Root cause (2026-05-20): Attempting to set review_value and review_confidence on 22 files using content.replace() in Node.js caused cascading corruption:

1. review_value: 7 followed by review_value:created: 2026-05-20 — review_value: matched the key but replacement concatenated the next line's value
2. review_confidence: 8 was placed OUTSIDE the frontmatter block (after ---)
3. review_value: 6provenance_state: extracted — two fields merged onto one line

Symptom: After a "successful" field-setting batch, lint shows 0 errors but read_file() reveals frontmatter like:

review_value: 7
updated: 2026-05-20

review_confidence: 8   ← outside frontmatter block!

Safe frontmatter update — Python line-by-line regex (preferred):

import os, re

for fn in files:
    fp = f"{entities_path}/{fn}"
    with open(fp, encoding='utf-8', errors='ignore') as f:
        content = f.read()
    
    fm_end = content.find('\n---', 4)
    body = content[fm_end + 4:]  # everything after first --- closing
    
    # Parse frontmatter line-by-line, skipping malformed
    fm_lines = content[:fm_end].split('\n')
    new_fm_lines = []
    skip_keys = {'review_value:created', 'review_confidence:created'}
    for line in fm_lines:
        stripped = line.strip()
        if any(sk in line for sk in skip_keys): continue
        if stripped.startswith('review_value:') and ':' in line:
            key = stripped[:stripped.index(':')].strip()
            if key == 'review_value': continue  # skip, will re-add
        if stripped.startswith('review_confidence:') and ':' in line:
            key = stripped[:stripped.index(':')].strip()
            if key == 'review_confidence': continue  # skip, will re-add
        new_fm_lines.append(line)
    
    new_fm_lines.append(f'review_value: {new_v}')
    new_fm_lines.append(f'review_confidence: {new_c}')
    
    new_content = '\n'.join(new_fm_lines) + body
    with open(fp, 'w', encoding='utf-8') as f:
        f.write(new_content)

NEVER use: content.replace('confidence: 0', f'confidence: {c}') in Node.js for frontmatter — it matches the first occurrence and corrupts surrounding context.

Node.js is acceptable only for single-field replacement where the field is on its own line and has a unique prefix:

content = content.replace(/^review_confidence:\s*.+$/m, `review_confidence: ${c}`);

But Python line-by-line processing is always safer.

Pattern 12: No-raw weak entities — sources: [] with value=3, confidence=7

Root cause (confirmed 2026-05-21): Entities created by extracting/conceptualizing content from OTHER entities' body citations (二手加工) receive review_value=3 (concept tier) and review_confidence=7. The frontmatter sources field is empty because the entity was not created from a raw article ingestion. Their actual provenance is buried in body citation markers (^[raw/articles/slug.md]).

Typical signature:

• review_value: 3, review_confidence: 7
• sources: [] (empty array, or field absent)
• Body contains ^[raw/articles/...] citation markers pointing to real raw articles
• Body length 1500–3000 chars with substantive analysis (NOT a stub)
• Original raw article found by searching body citations across the wiki

Tracing sources from body citations:

import os, re
entity_path = f"{wiki}/entities/{slug}.md"
with open(entity_path) as f:
    content = f.read()
citations = re.findall(r'\^\[raw/articles/([^)\]]+\.md)\]', content)
# For each citation, verify raw article exists and get title

Fix — add sources to frontmatter: Insert sources: [raw/slug] after the tags: line in frontmatter. Verify with git diff.

Scoring inflation: review_confidence=7 with sources: [] is inflated. Confidence ≤5 is more honest for extractive entities without a primary source.

Pattern 10b: INDEX DRIFT after bulk cleanup — always trust lint's tracked count

Pattern 16: 394 Entity Stubs Created from Orphan Raw — Future Expansion Targets (2026-05-21)

Root cause: After orphan raw cleanup (23 v2 duplicates deleted, 38 truncated RSS deleted, 41 missing frontmatter added), 782 orphan raw files remained. Of these, 394 had no corresponding entity and received minimal stub entities (body < 2KB, often just frontmatter + 1-2 bullet points).

Detection:

import os
entities_dir = "/Users/jinguo/wiki/entities"
stubs = []
for fn in os.listdir(entities_dir):
    if not fn.endswith('.md'): continue
    fp = f"{entities_dir}/{fn}"
    size = os.path.getsize(fp)
    if size < 2048:  # < 2KB
        stubs.append(fn[:-3])
print(f"Stubs < 2KB: {len(stubs)}")
# These include batch-created stubs from orphan raw + original thin entities

These are NOT errors — they are legitimate placeholder entities awaiting expansion. Priority targets for future expansion:

• Tag clusters: open-source: 46, aws-china-blog: 51, api: 44, gpt: 34

Strategy: Use wiki-entity-expansion skill's subagent workflow on high-value stubs (score ≥ 7, body < 1KB).

Pattern 17: Versioned Duplicate Raw Cleanup — v2/v3 without Base (2026-05-21)

Root cause: Raw articles re-ingested as -v2 or -v3 versions while the base version still existed (or didn't exist at all).

Decision matrix:

Scenario	Action
-v2 exists AND base exists	Delete the -v2 (duplicate)
-v2 exists AND base does NOT exist	Rename -v2 → base (promote)
-v3 exists AND base/-v2 exist	Delete -v3 (superseded)
-v3 exists AND only base exists	Delete -v3 (superseded)

2026-05-21 cleanup results:

• 23 v2 duplicates deleted (base existed)
• 7 orphan v2 → base promoted (no v1 existed)

Key entities affected:

• harness-engineering-让-coding-agent-可靠完成长程任务-v2.md → rewrote to reference base version

Detection:

import os, re
raw_dir = "/Users/jinguo/wiki/raw/articles"
versioned = {}
for fn in os.listdir(raw_dir):
    if not fn.endswith('.md'): continue
    m = re.match(r'(.+)-v(\d+)\.md$', fn)
    if m:
        base = m.group(1)
        version = int(m.group(2))
        if base not in versioned:
            versioned[base] = []
        versioned[base].append((version, fn))

for base, versions in versioned.items():
    base_exists = os.path.exists(f"{raw_dir}/{base}.md")
    print(f"{base}: base={'YES' if base_exists else 'NO'}, versions={sorted(v[0] for v in versions)}")

Pattern 18: Orphan Raw = Karpathy LLM-Wiki Leaf Node Design (2026-05-21)

Key design decision (confirmed 2026-05-21): 102 orphan raw files (3.5% of tracked pages) with 0 wikilinks are NOT a problem. Per Karpathy's llm-wiki design:

• raw/articles/ = source leaf nodes — source archives that don't need outbound links
• entities/ = synthesized nodes — reference raw via [[raw/articles/...]]
• Backlink from raw → entity (→ [[entities/...]]) was added where applicable

Acceptable orphan thresholds:

• raw/articles/: 3-5% orphans = acceptable (leaf nodes)
• entities/: < 50 orphans = acceptable, < 3 orphans = healthy
• concepts/queries/: 0 orphans = critical

Lint still tracks orphans — they appear as warnings, not errors. This is correct behavior.

Pattern 19: Tag Backfill — [article] Placeholder → Semantic Tags (2026-05-21)

Root cause: ~42 entities had tags: [article] — a placeholder tag with no classification value.

Tag taxonomy (5-category system):

• AI/ML: model, architecture, benchmark, training, inference
• People/Orgs: person, company, lab
• Techniques: optimization, fine-tuning, agent, mcp
• Meta: comparison, timeline, interview, job, tutorial
• Content: agent, ai, llm, anthropic, claude-code, security, aws

Fix:

# Replace [article] with semantic tags based on content analysis
import re
for fn in os.listdir(entities_dir):
    fp = f"{entities_dir}/{fn}"
    with open(fp, encoding='utf-8', errors='replace') as f:
        content = f.read()
    if 'tags: [article]' in content or 'tags:\n  - article' in content:
        # Infer semantic tags from content, filename, sources field
        new_tags = infer_semantic_tags(fn, content)
        # ... patch tags field

2026-05-21 results: 42 entities backfilled with semantic tags (agent/ai/llm/model/company/person/etc.)

Pattern 21: Ultra-Long Index Entry Lines — When Deduplication Requires Full Rebuild

Root cause: When entity titles contain escaped newlines (\\n) — common for long Chinese titles extracted from WeChat articles — the full entry line can be 500–2000+ characters. These ultra-long lines:

1. Make regex-based deduplication extremely fragile
2. Are invisible to grep for finding duplicates (the slug is buried mid-line)
3. Make patch operations fail because the old_string is too long to match reliably

Example: Line 437 of index.md contained a [[entities/claude-opus-47]] entry with a 1500-char title including embedded \\n\\n separators. The duplicate appeared at line 967.

Detection:

with open("index.md") as f:
    for i, line in enumerate(f, 1):
        if len(line) > 500:
            # Extract the slug from the long line
            import re
            m = re.search(r'\[\[entities/([^)|\]]+)', line)
            slug = m.group(1) if m else "UNKNOWN"
            print(f"Line {i} ({len(line)} chars): slug={slug[:60]}")

Fix: When deduplication involves ultra-long lines (≥500 chars), do NOT try to patch surgically. Use the rebuild from disk approach (Pattern 20):

# Instead of trying to remove duplicate line 967:
# 1. Read ALL entries from disk (correct source of truth)
# 2. Rebuild index with deduplication built in
# 3. Verify with lint

The disk scan naturally deduplicates because it reads from filenames, not from potentially-corrupt index entries.

When to apply: Deduplication with ≥3 duplicates, OR any entry line ≥500 chars, OR entries with \\n in the title field. Don't waste time on surgical patch attempts.

Pattern 22: Index Dedup Bug — Cross-Section vs Within-Section (2026-05-21)

Bug confirmed: When deduplicating index entries, the section-aware approach works correctly (only deduplicates within the same ## Section). However, a second pass using the same seen_per_section dict across sections caused entries from different sections to be treated as duplicates when they share the same slug (e.g., [[entities/X]] and [[raw/articles/X]] both exist for the same underlying article).

Symptom: After running dedup across all sections in one pass (not resetting seen_per_section per section), lint still reported DUPLICATE errors because:

• The section-aware pass correctly removed intra-section duplicates
• But the ultra-long-line dedup removed lines at WRONG line numbers (off-by-one from del lines[1543] + del lines[1542] where deletions shifted positions)

Safe dedup approach — always do it in two clean passes:

# Pass 1: Section-aware dedup (only deduplicates within same ## section)
lines = text.split('\n')
new_lines = []
current_section = None
seen_in_section = {}

for line in lines:
    m_section = re.match(r'^## (.+)$', line)
    if m_section:
        current_section = m_section.group(1)
        seen_in_section[current_section] = set()
        new_lines.append(line)
        continue
    m_entry = re.match(r'^-\ \[\[(entities|raw/articles)/([^\]|]+)(?:\|[^\]]+)?\]\]', line)
    if m_entry and current_section:
        uid = (m_entry.group(1), m_entry.group(2))
        if uid in seen_in_section[current_section]:
            print(f"  Dup in [{current_section}]: {uid[1][:50]}")
            continue  # skip duplicate
        seen_in_section[current_section].add(uid)
    new_lines.append(line)

# Pass 2: Ultra-long-line cleanup (separate pass, do NOT mix with dedup)
# See Pattern 21 for the ultra-long-line approach

Rule: Never combine section-aware dedup with ultra-long-line detection in the same pass. Each pass must be independently correct.

Pattern 23: Subagent Timeout on Index Repair — Scope Down or Use Python

Confirmed (2026-05-21): Delegating "fix all 26 index errors" to a subagent hits 600s timeout at ~39 API calls. Index repair requires:

• 1 call: lint to get error list
• 1 call: read index.md to identify error locations
• 1 call: Python/execute_code to compute fix plan
• 5-15 calls: apply fixes (patch operations)
• 1 call: write/index write
• 1 call: lint to verify
• 1 call: commit

That's ~11 calls minimum, but finding the exact line numbers for 26 errors across ghost+duplicate+missing categories needs ~20-40 calls depending on how many files need reading.

Fix: For index repair tasks with >10 errors:

1. Use Python/execute_code directly — Python can do all the logic (scan, dedup, ghost removal, missing insertion) in a single tool call
2. Or break into two subagent passes: Pass 1: audit only (lint → identify errors → report); Pass 2: fix based on Pass 1's findings
3. Or use disk-rebuild if errors >20 (see Pattern 20) — single atomic operation

Pattern 24: Bare Concept/Company Wikilinks — Mass Conversion Pattern (2026-05-21)

Root cause: Entity files frequently use bare company names and generic terms as wikilinks that don't exist as wiki entries:

• [[Agent]], [[LLM]], [[AI Agent]] → plain text
• [[Claude Code]], [[OpenClaw]], [[Codex]] → plain text
• [[Y Combinator]], [[GStack]], [[Conductor]] → plain text
• [[jq]], [[FFmpeg]], [[SQLite]] → plain text

Bulk conversion strategy:

# Pattern: bare terms that don't exist as wiki entries
bare_terms = [
    "Agent", "LLM", "AI Agent", "OpenClaw", "Claude Code", "Codex",
    "Y Combinator", "GStack", "Conductor", "RAG", "Hermes",
    "Stanford University", "Harvard University", "jq", "FFmpeg", "SQLite",
    "Agentic workflows", "Antigravity", "subagents", "interpretability tools",
    "SGLang", "vLLM", "iOS", "Android", "Google AI Edge Gallery",
    "memory management", "hallucination", "context window",
    "agent-harness", "context management", "agent-architecture",
    "skill-architecture", "natural language understanding",
    "brain-limb architecture", "contextual boundaries", "working-set",
    "prompt engineering", "deterministic execution",
    "agent-system-design", "agent-harness-engineering",
]

# These terms appear as [[Term]] (no prefix, no alias)
# They look like wikilinks but no entry exists → convert to plain text
for term in bare_terms:
    old = f"[[{term}]]"
    if old in content:
        content = content.replace(old, term)
        print(f"OK: {term}")

Special cases that ARE valid wikilinks — aliased ones with prefix:

• [[concepts/agentic-workflow-patterns|agentic workflows]] → keep (valid concept link)
• [[entities/claude-code|Claude Code]] → keep (valid entity link)
• [[entities/Agent Harness|Agent Harness]] → keep (valid entity link)

Footnote refs as wikilinks: [[^4]], [[^5]] → [^4], [^5] (remove brackets)

Pattern 25: macOS sed -i '' 's/.../.../' — Illegal Option Error

Bug: sed -i '' '...' on macOS fails with "illegal option -- A" when the command contains characters that the macOS sed interprets as flags.

Fix: Always use Python for complex sed operations involving Unicode or special characters:

# Instead of: sed -i '' 's/old/new/' file
with open(f) as fh: content = fh.read()
content = content.replace(old, new)
with open(f, 'w') as fh: fh.write(content)

Also: macOS cat -A doesn't support the -A flag. Use od -c or Python for debugging hidden characters.

Pattern 22: Index Rebuild After Sed/Patch Corruption — Multiple Rounds Required

When index.md has been modified by multiple patch/sed operations (e.g., removing ghosts, fixing duplicates, adding missing entries across several subagent passes), the file may accumulate overlapping corruption:

• Ghost entries removed but duplicates remain
• Missing entries added but some are duplicates of existing entries
• Total pages header drifts by 1-3

Fix strategy — three rounds of lint:

Round 1: Fix ghosts + duplicates
Round 2: Add missing entries
Round 3: Fix Total pages header drift

Each round: lint → identify remaining errors → fix → lint again.

When to rebuild from disk: If after 2 rounds of surgical fixes, errors remain >10, or if errors are a mix of MISSING + DUPLICATE + GHOST + INDEX DRIFT simultaneously, rebuild from disk is faster than iterative surgical fixes.

Subagent timeout pattern: When delegating index fixes to a subagent, it will likely hit max_iterations around 50 API calls. Index repair typically requires ~20-40 calls for:

• Reading the full index to identify all error types
• Computing the fix plan
• Applying fixes
• Verifying with lint

If the task scope includes both finding AND fixing all 26 index errors, it exceeds subagent capacity. Either:

1. Break into two subagent calls (audit + fix separately)
2. Use Python execute_code for all index operations
3. Use the disk-rebuild approach which is a single atomic operation

Use case: Find near-duplicate entity articles (80–100% content overlap) not caught by exact-hash dedup.

Algorithm: Extract plain text (strip frontmatter + markdown), tokenize into 5-word n-grams, compute Jaccard similarity. Pairs with Jaccard ≥ 0.60 are candidates.

Python implementation:

import os, re

WIKI = os.path.expanduser("~/wiki")

def extract_text(path):
    with open(path, encoding='utf-8', errors='ignore') as f:
        content = f.read()
    if content.startswith('---'):
        parts = content.split('---', 2)
        if len(parts) >= 3:
            content = parts[2]
    content = re.sub(r'```[\s\S]*?```', ' ', content)
    content = re.sub(r'`[^`]*`', ' ', content)
    content = re.sub(r'!\[[^\]]*\]\([^)]*\)', ' ', content)
    content = re.sub(r'\[[^\]]+\]\([^)]+\)', ' ', content)
    content = re.sub(r'[#*_~>`\-=|]', ' ', content)
    content = re.sub(r'\d+', ' ', content)
    content = re.sub(r'\s+', ' ', content).strip()
    return content.lower()

def get_ngrams(text, n=5):
    words = text.split()
    if len(words) < n:
        return set()
    return set(tuple(words[i:i+n]) for i in range(len(words) - n + 1))

def jaccard(a, b):
    if not a or not b:
        return 0.0
    inter = len(a & b)
    union = len(a | b)
    return inter / union if union else 0.0

files_data = []
for root, dirs, filenames in os.walk(WIKI):
    dirs[:] = [d for d in dirs if d not in ['.git', 'node_modules', '.trash']]
    for f in filenames:
        if not f.endswith('.md'):
            continue
        rel = os.path.relpath(os.path.join(root, f), WIKI)
        if 'raw/' in rel or '.trash/' in rel or not rel.startswith('entities/'):
            continue
        text = extract_text(os.path.join(root, f))
        ngrams = get_ngrams(text)
        if ngrams:
            files_data.append({'rel': rel, 'ngrams': ngrams})

duplicates = []
for i in range(len(files_data)):
    for j in range(i+1, len(files_data)):
        sim = jaccard(files_data[i]['ngrams'], files_data[j]['ngrams'])
        if sim >= 0.60:
            duplicates.append((files_data[i]['rel'], files_data[j]['rel'], sim))

duplicates.sort(key=lambda x: -x[2])
for a, b, sim in duplicates:
    print(f"{sim:.3f}: {a}\n      {b}")

Decision rules:

• 100% / near-100%: keep richer entity (better tags, review_value, provenance_state); delete the other
• 80–99%: same rule — prefer richer entity
• 60–79%: inspect both, decide case-by-case
• Slug variants (e.g., -1.md, hash suffix, truncation): almost always duplicates — delete the variant
• Malformed slug (e.g., oLTaFFlSocYKcWf253TurA.md): always delete — failed fetch artifact

Wikilink cascade fix after deletion — fix ALL layers before committing:

1. index.md: Remove entries for deleted files; update wikilinks from old → new slug
2. entities/concepts/comparisons: Replace [[entities/deleted-slug]] with [[entities/correct-slug]]
3. skills/: Same for skill SKILL.md files
4. raw/: Delete orphaned raw source files matching deleted entity slugs

Slug rename mapping example (2026-05-21):

fixes = [
    ("boris-cherny-devtools-ide-to-agent", "boris-cherny-新访谈开发工具正在从-ide-变成-agent-控制台"),
    ("claude-code-openclaw-memory-vector-db-doubt", "读完-claude-code-和-openclaw-的-memory-源码我对agent记忆需要向量数据库这件事产生了怀疑"),
    ("oLTaFFlSocYKcWf253TurA", "claude-opus-47"),
    ("ai-engineering-platform-aidlc-migration", "ai-驱动的大数据工程-从平台驱动到-aidlc-的范式迁移"),
    ("sap-unveils-the-autonomous-enterprise-1", "sap-unveils-the-autonomous-enterprise"),
    ("openai发布新一代实时语音模型-能够-2cb0c1", "openai发布新一代实时语音模型能够像人说话一样进行推理翻译和转录"),
    ("不改模型-不降质量-谷歌让gemma-4-d7989e", "不改模型不降质量谷歌让gemma-4快了3倍本地跑大模型彻底变天"),
]

PITFALL: log.md contains historical wikilinks to deleted files. Do NOT fix log.md — it's a historical record, not linted.

Pattern 20: Index Rebuild Pattern — When Incremental Fixes Fail (2026-05-21)

When to rebuild from disk: index.md has complex corruption (merged lines, duplicate entries, ghost entries) that incremental patching cannot safely fix.

Rebuild script (/tmp/rebuild_index.py):

import os, re, datetime

VAULT = "/Users/jinguo/wiki"
sections = {
    "Sources": ("raw/articles", r'\[\[raw/articles/([^|\]]+)'),
    "Entities": ("entities", r'\[\[entities/([^|\]]+)'),
    "Concepts": ("concepts", r'\[\[concepts/([^|\]]+)'),
    "Comparisons": ("comparisons", r'\[\[comparisons/([^|\]]+)'),
    "Queries": ("queries", r'\[\[queries/([^|\]]+)'),
}

lines = [
    "# Index\n",
    f"> Last rebuilt: {datetime.datetime.now().strftime('%Y-%m-%d %H:%M')} (disk scan)\n",
    # ... build entries from disk scan
]

total = 0
for section_name, (subdir, _) in sections.items():
    lines.append(f"\n## {section_name}\n")
    dir_path = os.path.join(VAULT, subdir)
    if os.path.exists(dir_path):
        files = sorted(f for f in os.listdir(dir_path) if f.endswith(".md"))
        total += len(files)
        for fname in files:
            slug = fname[:-3]
            title = slug
            try:
                with open(os.path.join(dir_path, fname)) as fh:
                    fm_text = fh.read(500)
                m = re.search(r'^title:\s*["\']?([^"\'\n]+)["\']?\s*$', fm_text, re.MULTILINE)
                if m:
                    title = m.group(1).strip()
            except:
                pass
            lines.append(f"- [[{subdir}/{slug}|{title}]]\n")

with open(os.path.join(VAULT, "index.md"), "w") as f:
    f.writelines(lines)

print(f"Rebuilt: {total} pages")

Verification: node scripts/wiki-lint.mjs . → 0 errors, Total pages header matches tracked count exactly.

When NOT to rebuild: Minor fixes (2-5 entries). Rebuild is for severe corruption only.

Fix:

actual=$(node scripts/wiki-lint.mjs . 2>&1 | grep -oP '\d+(?= tracked page)')
sed -i '' "s/Total pages: [0-9]*/Total pages: $actual/" index.md

Never count manually — tracked count includes all subdirectories.

Session Reference

• references/wiki-open-source.md — Wiki 开源准备：CC BY-NC-SA 4.0 许可 + 私人内容剥离方案A
• references/2026-05-audit-findings.md — Full audit findings, P0 fix log, and structural decisions from the 2026-05-20 open-sourcing preparation session.
• references/2026-05.21-comprehensive-overhaul.md — 2026-05-21 全面大修记录：394 entity stubs created, 782 orphan raw → classified, 23 v2 duplicates deleted, 7 v2→base promoted, 41 missing frontmatter added, lint 33→0, concepts 40→60, true orphans 10.4%→3.5%. 10 commits.
• references/2026-05-21-ghost-case-mismatch.md — Case-sensitivity ghost slug fix (prenorm vs preNorm). 33 errors → 0.
• references/2026-05-21-index-sync-session.md — Full session record: 7 entity deletions, 22 missing entities added to index, 21 missing raw sources added, all 0-error lint result.
• references/2026-05-20-95-entity-cleanup.md — 95-entity batch review: 48 non-AI/ML deletions, 47 kept.
• references/2026-05-21-frontmatter-fullscan.md — Frontmatter audit: 0 errors across ~2760 files.
• references/2026-05-22-78-entity-cleanup.md — Second-pass audit of 78 historical v×c<30 entities: 44 deleted, 33 kept. Documents ghost entry cascade, git reset --soft + force-with-lease rescue, push conflict resolution.
• references/2026-05-21-health-check.md — 2026-05-21 health check: 3→0 errors. Pattern 15 (phantom BROKEN LINK) discovered.
• references/2026-05-21-comprehensive-evaluation.md — 全面严苛评审（2026-05-21）：Lint 0 errors，880 raw orphans (64.9%)，17 thin entities，4 no-score entities，wikilink 密度分析，评分双峰分布，tag 使用率极低，concepts/queries 质量报告。附 wiki_audit.py + wiki_audit2.py 全量审计脚本。

Pattern 4: Bottom-block corruption with malformed markdown links

When patch tool corruption accumulates at the bottom of index.md, you may find a block like lines 2360-2375 containing:

1. - [Title](/entities/slug.md) — malformed Markdown link format (single space + no wikilink brackets)
2. Duplicate entity entries that belong in the Sources section
3. Possible ghost entries for non-existent files

Detection:

with open(os.path.join(VAULT, "index.md")) as f:
    lines = f.read().split('\n')
for i, l in enumerate(lines):
    if l.startswith(' - [') or l.startswith('  - ['):
        print(f"Line {i+1}: {l[:80]}")

Fix strategy — surgical Python removal:

with open(os.path.join(VAULT, "index.md")) as f:
    lines = f.read().split('\n')
# Identify range, verify, then:
new_lines = lines[:2359] + lines[2375:]
new_content = '\n'.join(new_lines)
with open(os.path.join(VAULT, "index.md"), 'w') as f:
    f.write(new_content)

Then re-add correct entries by reading titles from frontmatter and inserting before ## Bookmarks (for entities) or appending at the end (for queries).

Why patch fails here: The malformed block may have overlapping patterns (duplicates, ghosts, bad format). Python list slicing removes the entire block atomically.

index.md line corruption patterns

Index entries can be silently corrupted when a prior agent's patch replaces a line ending in ]] with content that lacks the closing brackets:

Pattern 1: Truncated wikilink — [[entities/foo followed by a newline (no closing ]]). The entry looks complete but is missing the link closing ]] and all content after the newline.

Pattern 2: Merged lines — Two entries get merged into one line when the first line's ]] is replaced with content that lacks ]].

Pattern 3: Section bleed — A contiguous block of lines from the wrong section gets inserted into another section. For example, a batch of raw/articles/... entries from the Sources section appear inside the Entities section, followed by a single malformed line containing both a raw entry and entity links concatenated together.

Detection: After any patch operation on index.md, re-check for entries longer than 200 chars (merged) or with wikilinks missing closing ]]. Always read the affected lines with repr() to see hidden newlines.

Fix order for Pattern 3 corruption:

1. First pass: scan for lines matching the pattern raw/articles/... - [[entities/ — split into separate lines
2. Second pass: deduplicate wiki link targets (use first occurrence, remove subsequent)
3. Third pass: fix Total pages header to match lint's tracked page(s) count

PITFALL: Multiple fix passes needed. The corruption often has overlapping patterns. After fixing merged lines, the deduplication pass may then expose duplicate entries that weren't visible before. After fixing duplicates, the Total pages header may still be wrong. Always re-run lint after each fix pass and repeat until 0 errors. Trust the lint's tracked count over the current header value.

Step 4: Audit frontmatter

import yaml

issues = []
for f in sorted(os.listdir(entity_dir)):
    if not f.endswith(".md"):
        continue
    with open(os.path.join(entity_dir, f)) as fh:
        content = fh.read()
    
    if not content.startswith("---"):
        issues.append(f"{f}: no frontmatter")
        continue
    
    try:
        parts = content.split("---", 2)
        fm = yaml.safe_load(parts[1])
        
        # Check required fields
        if "created" not in fm: issues.append(...)
        if "updated" not in fm: issues.append(...)
        if "type" not in fm: issues.append(...)
        if "tags" not in fm: issues.append(...)
        if "sources" not in fm: issues.append(...)
        
        # Check tag case
        if isinstance(fm.get("tags"), list):
            for t in fm["tags"]:
                if t and t != t.lower():
                    issues.append(f"{f}: tag '{t}' not lowercase")
    except Exception as e:
        issues.append(f"{f}: parse error: {e}")

Step 5: Fix common frontmatter bugs

Batch repair strategy

For large-scale frontmatter fixes (100+ files), use execute_code with a single Python loop over all files in all subdirectories. Process all issue categories in one pass to minimize file I/O:

for subdir in ["entities", "concepts", "raw/articles", "comparisons", "queries"]:
    for f in sorted(os.listdir(dirpath)):
        # Read → parse YAML → fix issues → write back
        # Fix ALL categories in one pass: missing fields, bad types,
        # uppercase tags, sources .md suffix, etc.

Critical reassembly pattern:

# CORRECT: ensures newline after opening ---
new_content = f"---\n{raw_fm.lstrip(chr(10))}\n---{body}"

# WRONG: may produce ---title: (no newline = broken YAML)
new_content = f"---{raw_fm}\n---{body}"

YAML titles with colons — CRITICAL

Titles containing colons break YAML parsing even when the colon is inside a quoted string. The standard yaml.safe_load(parts[1]) approach fails because YAML parsers see title: "A²RD: as a mapping key title with value "A²RD" followed by an unexpected :. This is a common pattern with English titles.

Correct parsing approach — always wrap the entire frontmatter as a document:

parts = content.split("---", 2)
fm = yaml.safe_load(f"---\n{parts[1]}\n---")  # ✓ Correct
# yaml.safe_load(parts[1])  # ✗ Fails on titles with colons

Fix by quoting the title value:

title: "A²RD: Agentic Autoregressive Diffusion for Long Video Consistency"
title: "AI-powered honeypots: Turning the tables on malicious AI agents"
title: "Bitcoin news: Soon, traders will be able to bet on BTC volatility"

Without quotes, any title with : followed by a space is parsed as a YAML mapping pair instead of a string value.

YAML titles with inner double quotes (Chinese)

Chinese text often uses " for emphasis (e.g. "潜规则"). When the title is wrapped in double quotes and contains inner ", YAML breaks:

# BROKEN: closing " at 潜规则 ends the YAML string prematurely
title: "从 0 到 1 教你写 Agent Skill，让 AI 懂你的"潜规则""
# FIX: use single-quote wrapping, escape any inner single quotes as ''
title: '从 0 到 1 教你写 Agent Skill，让 AI 懂你的"潜规则"'

Unclosed double quotes in publisher field

WeChat-sourced articles often have publisher: "微信公众号 - xxx with an opening " but no closing ". This silently breaks YAML parsing. Always check for odd quote counts per line.

---<no-newline> frontmatter corruption

Critical bug pattern: When reassembling frontmatter after content.split("---", 2), using f"---{raw_fm}\n---{body}" produces ---title: (no newline after ---) if raw_fm has been .strip()'d or .lstrip('\n')'d. This corrupts YAML parsing for the entire file.

Detection: Files where content[3:4].isalpha() after --- (no newline separator). Fix: Replace ---<word> with ---\n<word> at file start.

Prevention: Always use f"---\n{raw_fm.lstrip(chr(10))}\n---{body}" when reassembling, ensuring a \n after the opening ---.

Field-merge corruption when patching multiple frontmatter fields

NEW pitfall (2026-05-20): When adding TWO missing fields to a frontmatter in sequence (e.g., first review_value, then provenance_state), appending each without explicit \n causes them to merge:

# BEFORE: confidence: 0.9 (last proper field)
# STEP 1: append review_value: 6 (no \n after, just replaced)
# STEP 2: append provenance_state: extracted (no \n after)
# RESULT:
confidence: 0.9
review_value: 6provenance_state: extracted
---     ← closing --- merges with '6', becomes '6---'

Lint reports NO FRONTMATTER or YAML ERROR but the file visually looks fine.

Detection: If lint reports NO FRONTMATTER on a file that clearly has --- markers, the frontmatter has field-merge corruption. Verify with:

with open(f) as fh:
    content = fh.read()
import re
merged = re.findall(r': [^\n]{1,50}[a-z][a-z_]+:', content)
print("Potential merged fields:", merged)

⚠️ Pitfall: Overly broad field-merge regex breaks 2700+ files (2026-05-20) Do NOT use patterns like review_value:\d+created: to detect field merges — "created" appears in URLs and wikilinks throughout the wiki, so this matches thousands of false positives.

Safe detection approach — scan YAML directly:

import yaml, re

for f in wiki.rglob("*.md"):
    content = f.read_text(errors='replace')
    if content.count('---') < 2:
        continue
    parts = content.split('---', 2)
    if len(parts) < 3:
        continue
    # Wrap in YAML document format to handle titles with colons
    try:
        fm = yaml.safe_load(f"---\n{parts[1]}\n---")
    except:
        print(f"YAML ERROR: {f.name}")
        continue
    # Detect merged fields: two field names smushed together
    raw_lines = parts[1].split('\n')
    for line in raw_lines:
        m = re.match(r'^(\w+): (.+)$', line)
        if m:
            val = m.group(2)
            # Check if value ends with another field name pattern
            if re.search(r'[a-z][a-z_]+:$', val):
                print(f"MERGED: {f.name}: {line[:60]}")

Safe fix — per-file YAML round-trip:

import yaml, re

for f in wiki.rglob("*.md"):
    content = f.read_text(errors='replace')
    if content.count('---') < 2:
        continue
    parts = content.split('---', 2)
    if len(parts) < 3:
        continue
    raw_fm = parts[1]
    body = parts[2]
    try:
        fm = yaml.safe_load(f"---\n{raw_fm}\n---")
    except:
        continue
    # Rebuild cleanly
    lines = []
    for k, v in fm.items():
        if isinstance(v, list):
            lines.append(f"{k}:")
            for item in v:
                lines.append(f"  - {item}")
        else:
            lines.append(f"{k}: {v}")
    new_fm = '\n'.join(lines)
    new_content = f"---\n{new_fm}\n---{body}"
    f.write_text(new_content)

Fix: Rebuild the frontmatter from scratch:

with open(f) as fh:
    raw = fh.read()
idx1 = raw.find('---')
idx2 = raw.find('---', idx1 + 3)
body = raw[idx2+3:]
fm = "---\ntitle: ...\ntype: entity\nreview_value: 7\n---\n"
with open(f, 'w') as fh:
    fh.write(fm + body)

Prevention: Always add \n explicitly after each field when patching. Never bare string concatenation. When in doubt, always rebuild frontmatter from scratch rather than patching inline.

Tags with # prefix in flow sequences

tags: [#Anthropic, #MCP, #Claude-Code] — the # prefix inside [] flow sequences is invalid YAML. Strip # from all tags:

raw_fm = re.sub(r'tags:\s*\[([^\]]+)\]',
    lambda m: 'tags: [' + re.sub(r'#(\S+)', r'\1', m.group(1)) + ']',
    raw_fm)

Unquoted URLs in source/source_url fields

source_url: https://mp.weixin.qq.com/s/xxx — the :// in URLs breaks YAML parsing (colon in value). Always quote URLs:

raw_fm = re.sub(r'^(source(?:_url)?):\s+(https?://\S+)$', r'\1: "\2"', raw_fm, flags=re.MULTILINE)

Tags with # prefix

Some pages have - #claude-code instead of - claude-code. Strip the # prefix:

patch(old="- #claude-code", new="- claude-code")

Mixed-case tags

Use patch to lowercase:

patch(old="Claude-Code", new="claude-code")

sources with .md suffix

Wikilinks in sources: field should not have .md suffix: sources: [raw/articles/foo.md] → sources: [raw/articles/foo]. Batch fix with regex on raw frontmatter.

Duplicate sources field

Some pages have sources: [] twice due to prior merges:

patch(old="sources: []\nconfidence: medium\nsources: []", new="sources: []")

Missing fields

For batches of similar fixes (e.g., adding empty sources: [] to standalone pages), use execute_code with Python file I/O:

for slug in no_sources:
    path = os.path.join(entity_dir, slug + ".md")
    with open(path) as f: content = f.read()
    parts = content.split("---", 2)
    new_fm = parts[1].rstrip() + "\nsources: []\n"
    with open(path, 'w') as f: f.write(content.replace(parts[1], new_fm))

Step 6: Rotate log.md

When log.md exceeds 500 entries:

cd <wiki_root>
cp log.md log-2026-04.md

Then write a fresh log.md with a rotation notice and the current action.

Git Push Rescue Pattern (2026-05-22)

When a cleanup commit fails to push because local branch is behind remote:

# State: c1b3afc1 committed, but git push → "tip of branch is behind"
# Solution:

# 1. Soft-reset to align local HEAD with remote (preserves staging area)
git reset --soft origin/main

# 2. Verify staged files are correct (should be your cleanup changes)
git status --short | wc -l   # ~382 files for 44 deletions + broken link fixes

# 3. Force-push with lease (safe: refuses if remote advanced)
git push --force-with-lease

Why --force-with-lease instead of --force: It checks that no one else pushed to the remote branch since your last fetch. If the remote advanced, the push is rejected instead of overwriting their changes.

Why --soft: Resets HEAD to match origin/main but keeps all changes in the staging area. Your cleanup commit is preserved as staged changes, ready to push.

Step 7: Update SCHEMA.md

Check if SCHEMA.md is missing:

• Review metadata definitions (review_value/review_confidence/etc.)
• Wikilink format convention ([[subdir/page-slug]])
• 原文存档 backlink requirement
• Tag lowercase rule

If missing, update SCHEMA.md with these additions.

Step 8: Run lint and verify

After all fixes, run the linter:

node scripts/wiki-lint.mjs <wiki_root> 2>&1

Target: 0 errors. Warnings (MISSING sha256, NO FRONTMATTER on raw articles) are acceptable debt.

Pitfall: Citation regex matching frontmatter sources: arrays

Bug (confirmed 2026-05-22): CITATION_RE pattern /\^\\[([^\\]]+)\\]/g with [^\\]]+ greedily matches through commas inside YAML flow sequences.

Example: sources: [raw/articles/foo.md, raw/articles/bar.md] — the , between two paths is matched as part of the citation, producing raw/articles/foo.md, raw/articles/bar.md as a single broken path.

Fix: Use [^\\],]+ instead of [^\\]]+ in all citation regexes:

// WRONG — comma in sources array breaks the match
const CITATION_RE = /\^\[([^\]]+)\]/g;
const SIMPLE_CIT_RE = /\^\\[([^\]\.]+\.md)[^\]]*\\]/;

// CORRECT — stops at comma
const CITATION_RE = /\^\[([^\],]+)\]/g;
const SIMPLE_CIT_RE = /\^\\[([^\]\.]+\.md)[^\],]*\\]/;

Verification after fix: Lint should report 0 BROKEN CITATION errors for frontmatter sources arrays.

Known lint false positives

WL_RE False Positive: Python List/Dict Literals (Enhanced — confirmed 2026-05-22)

Symptom: BROKEN LINK: entities/X -> [["macd", "rsi_14", ...]] — a Python list inside a code block is matched as a wikilink.

Root cause: The wikilink regex \[\[(.+?)\]\] matches [[...]] anywhere in the file, including inside Python/JavaScript code. Even wrapping in list() or .loc does NOT help — the inner [[...]] still matches.

Fix levels (escalate if lint still fails after each attempt):

# Level 1: list() wrapper — FAILS (inner [[...]] still matches)
indicators = list(stock_df.columns[["macd", "rsi_14", ...]])

# Level 2: .loc accessor — FAILS
indicators = stock_df.loc[:, ["macd", "rsi_14", ...]]

# Level 3: Column index range — SAFE (no [[...]])
indicators = list(stock_df.columns[
    stock_df.columns.get_loc("macd"):stock_df.columns.get_loc("close_200_sma")+1
])

# Level 4 (safest): String literal list — no brackets at all
indicators = ["macd", "rsi_14", "boll", "boll_ub", "boll_lb", "close_50_sma", "close_200_sma"]

Verification: After each fix, run node scripts/wiki-lint.mjs 2>&1 | grep "error\(s\)" until 0 errors.

Also applies to JavaScript: obj[['key1', 'key2']] → restructure to avoid [[...]].

Chinese filename path false positive (macOS Node.js encoding bug): Lint may report BROKEN CITATION for raw article paths containing Chinese characters, even when os.path.exists() confirms the file exists. Root cause: Node.js existsSync on macOS fails to resolve UTF-8 encoded Chinese filenames. The file is always on disk — this is a lint false positive. Ignore it.

Wiki-lint "broken citation" false positives — line numbers misread as filenames: Lint uses a regex like ([^/,]+) to extract citation source filenames. This pattern incorrectly matches line numbers at the end of a citation. Example: ^[raw/articles/ai-agent-tool-count-trap.md:65-66,99] — the ,99 is a line number (single line 99), NOT a second source file. The lint reports this as "broken citation: ai-agent-tool-count-trap.md, raw/articles/99" which is wrong. Before "fixing" anything, verify with:

import re
citation = "ai-agent-tool-count-trap.md:65-66,99"
parts = citation.split(',')
is_false_positive = bool(re.match(r'^\d+$', parts[1])) if len(parts) > 1 else False

Wikilink .md suffix is a real error — fix both raw/articles/ and entities/ patterns: Wikilinks with .md suffix cause BOTH lint errors AND break Obsidian link resolution. Always fix these — they are real errors:

# Fix [[raw/articles/slug.md]] → [[raw/articles/slug]]
content = re.sub(
    r'\[\[(entities|raw|concepts|comparisons|queries)/([^)|\]]+)\.md(\|[^]]*)?\]\]',
    lambda m: f'[[{m.group(1)}/{m.group(2)}{m.group(3) or ""}]]',
    content
)

date: field is deprecated — replace with created:

content = re.sub(r'^date:\s*(.+)$', r'created: \1', content, flags=re.MULTILINE)

Ghost wikilink pattern — wrong slug in index entry

"GHOST: index entry [[entities/slug-X]] has no file on disk" means the wikilink slug in index.md doesn't match any actual filename. Fix: grep -n "slug-X" index.md to find the bad entry, then ls entities/*foo* to discover the actual filename, then patch to correct.

Batch YAML frontmatter repair patterns (2026-05-20 session)

This session fixed 61 frontmatter issues across 1320 entities in one coordinated pass:

Issue	Count	Fix
Missing type field	21 files	Add type: entity
Missing review_value field	40 files	Add review_value:
sources field has .md suffix	32 files	Strip .md suffix
Duplicate entity_type field (also has type)	52 files	Remove entity_type, keep type
YAML corruption via patch `	- ` prefix	13 files

Patch tool |- prefix corruption (NEW — critical): The patch tool, when applied to frontmatter blocks, inserts |- prefix at the start of list items (e.g., |- tag1) instead of - . This corrupts YAML.

Detection:

grep -n "^- |" entities/*.md

Should return zero matches.

Fix (one-liner):

cd ~/wiki && sed -i '' 's/^|- /- /' entities/*.md && sed -i '' 's/^|- /- /' concepts/*.md

Run after ANY patch operation that touched frontmatter blocks. Verify: grep -c "^- |" entities/*.md should return all zeros.

Prevention: For frontmatter list items, use patch with exact lines that already start with - , not |- . If the original line has - tag1, match - tag1 exactly.

Orphan entity repair — systematic method (2026-05-20)

When lint reports "True orphans" among entities (0 out-links + 0 in-links), fix by adding 3-6 related entity wikilinks in the ## 深度分析 section:

For entities/: real content pages

with open(f"{entities_dir}/{slug}.md") as f:
    content = f.read()

if "## 相关页面" in content:
    # Append related links to existing section
    pass
else:
    insert_text = "\n\n## 相关页面\n\n" + "\n".join(f"- [[entities/{rel}]]" for rel in related)
    content = content.rstrip() + insert_text

with open(fpath, 'w') as f:
    f.write(content)

For concepts/: stub pages with "> Stub page" markers — these are placeholder pages with minimal content and no real wikilinks:

1. Remove the > Stub page — 内容待补充 block entirely
2. Find REAL entity files that match the concept topic using disk scan:

   existing = [n for n in os.listdir('entities') if 'mcp' in n.lower()]
   

3. Replace the stub block with a ## 相关页面 section containing real entity links

⚠️ common pitfall: A concepts/ orphan may already have ## 相关页面 section but still report as orphan — because the section contains only > Stub page text with zero actual [[wikilinks]]. The fix is to replace the stub content with real links, not to add a duplicate section.

Truncated slug entities — detection and fix

NEW (2026-05-20): Some entity filenames are truncated at system boundaries (e.g., from-ssh-to-rest-...-e ending mid-word). This causes:

• Duplicate entities: the truncated slug version AND a full-slug version referencing the same raw article
• Lint MISSING type in frontmatter errors despite type: field existing (frontmatter YAML is malformed due to field-merge corruption)

Detection:

# Files ending with single-char hyphen = truncated slug
ls entities/ | grep -E '[a-z]-$'

Pattern: When the raw article slug is longer than the filesystem limit, the ingestion system truncates and adds a trailing hyphen. Both the truncated entity and full entity exist.

Fix strategy: Compare the two versions. The truncated version typically has much more content (14-22KB, full body) while the full version is smaller (3-4KB, summary). Keep the truncated version as the primary entity, delete the full version, and update all references.

# Find both versions of the entity
import os, subprocess

truncated_slug = "announcing-aws-cdk-mixins-composable-abstractions-for-aws-re"
full_slug = "announcing-aws-cdk-mixins-composable-abstractions-for-aws-resources-amazon-web-s"

trunc_path = f"entities/{truncated_slug}.md"
full_path = f"entities/{full_slug}.md"

trunc_size = os.path.getsize(trunc_path) if os.path.exists(trunc_path) else 0
full_size = os.path.getsize(full_path) if os.path.exists(full_path) else 0

print(f"Truncated: {trunc_size} bytes, Full: {full_size} bytes")
# Keep the larger one as primary entity

UPPER-TAG warnings — acceptable debt

95 UPPER-TAG warnings remain after full audit. These are wikilinks where the link target doesn't match the actual filename casing. These don't break functionality and fixing them is tedious. Accept this debt unless doing a dedicated wikilink normalization sprint.

Batch entity quality classification (tiering before bulk operations)