| name | wiki-maintenance |
| description | Comprehensive wiki operations on the Obsidian wiki at ~/wiki — routine maintenance (frontmatter, tags, lint, orphans, git), content expansion (concepts, entities), batch quality sprints, and crisis repair (YAML corruption, citation gaps, duplicate deduplication). |
| category | wiki |
Wiki Maintenance Playbook
Scope
Routine maintenance operations on the Obsidian wiki at ~/wiki (tech wiki) or ~/wiki-life (life wiki): frontmatter fixes, tag cleanup, lint validation, orphan detection, git housekeeping.
Prerequisites
# For tech wiki:
cd ~/wiki
# For life wiki:
cd ~/wiki-life
Lint Validation (always run after any change)
Tech wiki (~/wiki)
node scripts/wiki-lint.mjs .
# Expected: 0 error(s). Warnings are informational.
Life wiki (~/wiki-life)
No lint script exists — validation is manual via Python execute_code or shell inspection:
# Check frontmatter completeness
for f in entities/*.md concepts/*.md comparisons/*.md cases/*.md queries/*.md; do
head -10 "$f" | grep -q "^type:" || echo "MISSING type: $f"
head -10 "$f" | grep -q "^created:" || echo "MISSING created: $f"
head -10 "$f" | grep -q "^updated:" || echo "MISSING updated: $f"
head -10 "$f" | grep -q "^tags:" || echo "MISSING tags: $f"
done
# Check index.md stats vs actual
for d in entities concepts cases comparisons queries reviews "raw/articles"; do
count=$(ls $d/*.md 2>/dev/null | wc -l | tr -d ' ')
echo "$d: $count"
done
Skills Sync — Multi-Category rsync
Skills live in multiple category subdirectories under ~/.hermes/skills/: wiki/, creative/, devops/, software-development/, apple/, autonomous-ai-agents/, mcp/, etc. The wiki's skills/ directory is the union of all categories, synced via rsync.
Pitfall: Single-Category rsync + --delete Deletes Other Categories
Symptom: After syncing ~/.hermes/skills/wiki/ → ~/wiki/skills/ with --delete, skills like anti-ai-style-library, design-md, comfyui etc. disappear from ~/wiki/skills/ because they live in ~/.hermes/skills/creative/, not ~/.hermes/skills/wiki/.
Wrong (only syncs wiki-category skills, deletes everything else):
rsync -av --delete ~/.hermes/skills/wiki/ ~/wiki/skills/
Correct — sync ALL categories, one rsync per source dir without --delete (use --delete only on the first/primary source):
# Primary sync (wiki category, with --delete to remove stale files)
rsync -av --delete ~/.hermes/skills/wiki/ ~/wiki/skills/
# Additive syncs (other categories, NO --delete — they only add, never remove wiki-category files)
for cat in creative devops software-development apple autonomous-ai-agents mcp productivity research media data-science github note-taking smart-home social-media red-teaming mlops gaming email; do
src="$HOME/.hermes/skills/$cat/"
[ -d "$src" ] && rsync -av "$src" ~/wiki/skills/
done
Why no --delete on secondary categories: If creative skills sync with --delete, it would remove wiki-category files that don't exist in creative/. The --delete flag means "make the destination look exactly like the source" — removing anything not in the source.
Safer alternative — sync the parent directory directly (includes all categories):
# This copies ALL skill categories at once, --delete is safe here
rsync -av --delete ~/.hermes/skills/ ~/wiki/skills/ --exclude='*.pyc' --exclude='__pycache__'
Caveat: This also copies non-category files (e.g., DESCRIPTION.md at the top level). Verify with git status after sync.
Recovery if skills were deleted: Re-run the correct rsync command, then git add skills/ && git commit.
Nested .git Directories in Skills
Skill dirs under ~/.hermes/skills/ must NOT contain .git/ subdirectories — Git clients discover them and display phantom repos. See references/nested-git-repo-cleanup.md for diagnosis and fix.
Common Operations
Entity Census — Pitfall: -maxdepth 1 Misses Subdirectory Files
When counting entities in entities/, always use recursive find. The entities/ directory contains both top-level .md files AND subdirectories (e.g., entities/1-million-exposed-ai-services-hackernews/) that hold additional .md files.
Wrong (undercounts by ~300 files):
find ~/wiki/entities -maxdepth 1 -type f -name "*.md" | wc -l
# Returns ~2450 (misses subdirectory files)
Correct:
find ~/wiki/entities -type f -name "*.md" | wc -l
# Returns ~2767 (includes all levels)
Breakdown (as of 2026-06-25):
- Top-level
.md files: ~2,450
- Subdirectory
.md files: ~317
- Total recursive: ~2,767
When user asks "how many entities": Always report the recursive count. The subdirectory files are real entities (often paired with asset dirs), not noise.
Wikilink Graph Analysis & Knowledge Hub Discovery
When analyzing wiki structure (e.g., for book planning, content audit, or MOC creation), build a cross-reference graph from wikilinks:
import os, re
from collections import defaultdict
entities_dir = os.path.expanduser("~/wiki/entities")
link_to = defaultdict(set) # slug -> set of slugs linking to it
for fname in sorted(os.listdir(entities_dir)):
if not fname.endswith('.md') or not os.path.isfile(os.path.join(entities_dir, fname)):
continue
with open(os.path.join(entities_dir, fname), 'r', encoding='utf-8') as f:
content = f.read()
slug = fname.replace('.md', '')
wikilinks = re.findall(r'\[\[(?:entities/)?([^\]|]+?)(?:\|[^\]]+)?\]\]', content)
for link in wikilinks:
link_slug = link.strip().split('/')[-1]
if link_slug != slug:
link_to[link_slug].add(slug)
# Top knowledge hubs (highest in-degree)
top_hubs = sorted(link_to.items(), key=lambda x: -len(x[1]))[:20]
for slug, refs in top_hubs:
print(f" {len(refs):3d} refs {slug}")
What the graph reveals:
- Hubs (high in-degree): Knowledge anchors — use as chapter/section anchors when structuring books or MOCs
- Orphans (0 in + 0 out): Truly disconnected
- Domain clusters: Tag co-occurrence + link density reveals topic boundaries
See: references/wiki-knowledge-graph-analysis.md for full book-structuring methodology.
Frontmatter Tag Cleanup
Problem: Files with tags: [] (empty array) or duplicate tags: YAML keys (last-value-wins — empty array overwrites real tags).
Detection:
import os, re
wiki_path = "entities"
empty = []; dup = []
for root, dirs, files in os.walk(wiki_path):
for f in files:
if not f.endswith('.md'): continue
fp = os.path.join(root, f)
with open(fp) as fh: content = fh.read()
fm_match = re.search(r'^---\n(.*?)\n---', content, re.DOTALL)
if not fm_match: continue
fm = fm_match.group(1)
if re.search(r'^tags:\s*\[\s*\]', fm, re.MULTILINE): empty.append(fp)
if fm.count('tags:') > 1: dup.append(fp)
Fix empty arrays — keyword-based auto-tagging:
import re
def auto_tag(title, body, source):
ti = title.lower(); bd = body.lower(); s = source.lower()
kw_map = [
('nvidia', ['nvidia', 'gpu kernel', 'cuda']),
('claude-code', ['claude']),
('anthropic', ['anthropic']),
('security', ['security', 'hack', 'vulnerability', 'supply chain', 'exploit', 'malicious', 'phishing', 'fuzzing', 'cve']),
('agent', ['agent']),
('multi-agent', ['multi-agent', 'multiagent', 'multi agent']),
('harness-engineering', ['harness']),
('rag', ['rag', 'retrieval']),
('memory', ['memory']),
('mcp', ['mcp']),
('llm', ['llm', 'language model', '大模型', 'gpt', 'gemma', 'deepseek', 'minimax']),
('skill', ['skill']),
('openclaw', ['openclaw']),
('aws', ['aws', 'amazon', 'bedrock', 'sagemaker']),
('open-source', ['open-source', 'open source', '开源']),
('video', ['video']),
('inference', ['inference', '推理', 'speculative', 'kv cache']),
('evaluation', ['benchmark', '评估', '评测']),
('architecture', ['architecture', '架构']),
('engineering', ['engineering', '工程']),
('ai', ['ai', '人工智能']),
('news', ['newsletter', 'venturebeat', 'cio dive', 'theregister', 'hackernews']),
]
t = []
for tag, keywords in kw_map:
for kw in keywords:
if kw in ti or kw in bd: t.append(tag); break
if not t:
if 'wechat' in s or '微信' in ti: t = ['wechat', 'ai']
elif 'newsletter' in s or 'rss' in s: t = ['newsletter', 'ai']
else: t = ['article', 'ai']
seen = set(); unique = []
for x in t:
if x not in seen: seen.add(x); unique.append(x)
return unique[:5]
for fp in empty:
with open(fp) as fh: content = fh.read()
# Extract metadata, call auto_tag(), then:
new_content = re.sub(r'^tags:\s*\[\s*\]', f"tags: [{', '.join(tags)}]", content, count=1, flags=re.MULTILINE)
with open(fp, 'w') as fh: fh.write(new_content)
Entity Title Quality Scan
Entity titles that are just the slug (hyphens/no spaces), hash IDs, or all-lowercase are unhelpful. Use references/entity-title-quality.md for detection script, three fix categories (lowercase → title-case, hyphenated → Chinese title from content, hash → descriptive title), and batch fix patterns.
Patch Tool |- Prefix Corruption
After ANY patch tool edit (not just frontmatter), check the affected lines. The patch tool can corrupt YAML list item prefixes in BOTH frontmatter AND body lists (|- instead of - ).
Frontmatter (YAML parse failure):
sed -i '' 's/^|- /- /' entities/*.md
Body Related entities lists (visible corruption — multiple items merged into one line):
# Example: "- [[entities/A]]- [[entities/B]]- [[entities/C]]" merged into one list item
content = content.replace(
'- [[entities/X]]- [[entities/Y]]- [[entities/Z]]',
'- [[entities/X]]\n- [[entities/Y]]\n- [[entities/Z]]'
)
Always inspect list sections after patching Related entities or other - lists.
CRITICAL PITFALL — don't sed-bulk-fix ^|- in index.md blindly (verified 2026-06-18):
The sed -i '' 's/^|- /- /' index.md recipe is unsafe in the merged-line case. When patch tool corrupts an index.md line, it often merges 2-3 valid [[wikilink|display]] entries into ONE physical line. That merged line literally starts with |- (because patch inserted a list prefix into the middle of a line that also contains the merged entries' first entry). Running the sed recipe DELETES THE ENTIRE MERGED LINE — losing 2-3 real entries per affected line. Verified in a 4560-page index: a naive sed pass would have silently destroyed 3 entries.
Symptom of merged-line corruption (NOT a simple prefix swap):
- [[entities/A]]描述...- [[entities/B]]描述...
- [[raw/X|...]]\n- [[raw/Y|...]]可靠性、上下文与恢复机制]]
Two real entries live on one physical line. There is NO \n between them — sed 's/^|- /- /' matches the leading |- and the line is "fixed" — by being deleted.
Detection (BEFORE running sed) — use awk or rg, NOT BSD grep -E:
BSD grep -E '^\|- ' on macOS has been observed to return wildly wrong counts (e.g. reporting 2521 "corruptions" when actual count is 0). The ^ anchor in -E mode combined with \| (escaped pipe) interacts badly. Don't trust it for this check.
# CORRECT — ground truth count of corrupt lines
awk '/^\|- / {c++} END {print c}' index.md
# Or:
rg -n '^\|- ' index.md | wc -l
If awk reports 0 → no fix needed, do not run sed. If awk reports N > 0 → manual inspection, NOT bulk sed.
Correct fix — patch tool with old_string/new_string on the exact merged line:
- Run
rg -n '^\|- ' index.md to get line numbers of the small set of actually-corrupt lines (typically 1-5, not hundreds)
- For each corrupt line, view 2 lines of context with
sed -n '$((line-1)),$((line+1))p' index.md
- Use
patch tool with the EXACT merged-line content as old_string, and the correctly split version as new_string:
old_string: - [[entries/A]]描述- [[entries/B]]描述
new_string: - [[entries/A]]描述
- [[entries/B]]描述
- Re-run lint:
node scripts/wiki-lint.mjs . — expect MISSING from index: ... errors to drop to 0.
Why this matters in practice: The wiki has a pre-commit wiki quality gate hook that runs lint and blocks commits with MISSING from index errors. Bulk-sed destroying merged lines produces silent data loss that only surfaces in next audit.
Linter false positive caveat: wiki-lint.mjs reports MISSING from index: <entity-slug> for any tracked entity that has no [[entities/<slug>|...]] entry in index.md. When a merged line is "fixed" by sed-deletion, the second entry's slug is now missing → lint fails. Always cross-check lint error count == 0 after the fix.
Verify Report Claims Against Disk Before Commit (2026-06-18)
Pattern: When a report (REVIEW.md, audit report, LLM-generated summary, batch-fix completion report) claims specific counts ("1,032 entities updated", "522 broken links removed", "29 weak entities promoted"), DO NOT trust the numbers at face value — verify with ground-truth probes:
# Ground-truth the modified-file count
git diff --shortstat
# → "68 files changed, +355/-600"
# Ground-truth the modified file list
git diff --name-only
# Ground-truth the actual lint state
node scripts/wiki-lint.mjs /Users/jinguo/wiki 2>&1 | tail -10
# → expect "── Errors (N) ──" with count
# Ground-truth the modified line volume per file
git diff --stat | sort -t'|' -k2 -n | tail -10
Real session 2026-06-18: A "Wiki 整体修复完成报告" claimed "1,032 entities 补段落引用". Ground-truthing with git diff --shortstat showed only 68 files changed (+355/-600). The 1,032 number was LLM hallucination. The work itself was real (broken-link cleanup, MOC additions, citation backfill) — but the count was off by 15×. The pre-commit wiki quality gate would have caught missing citations in a different way (EXCESS INFERRED warnings), so the actual fix was valid — but the report was untrustworthy as a basis for the commit message.
Rule: Before commit, do at least these three checks against any batch-fix report:
git diff --shortstat → confirm modified-file count is in the ballpark of claimed numbersnode scripts/wiki-lint.mjs . → confirm error count matches claims (typically 0)- Spot-check 3-5 modified files via
git diff <path> → confirm changes match report description
If any check fails the smell test, REWRITE the commit message to use the actual stats, not the claimed ones. The user will see the commit message — accuracy matters more than the report.
UTF-8 Byte Count vs Char Count — lint EXCESS INFERRED Inflation
Symptom: wiki-lint.mjs reports EXCESS INFERRED: entities/SLUG has N/N (100%) where N is wildly inflated (e.g., 368 for a 530-line file). Manual verification shows 0 citations, no actual self-links.
Root cause: wiki-lint.mjs reads files as raw bytes and uses byteLength / 50 to estimate paragraph count. Files containing emoji (🔥) or CJK Extension-B/C characters encode each character as 4 bytes instead of 3. This doubles the byte count and thus doubles the estimated paragraph count.
Detection:
import os
# Compare byte count vs char count for suspect files
fp = 'entities/十年老技术开发的-ai-agent-探索之路.md'
with open(fp, 'rb') as f: raw_bytes = len(f.read())
with open(fp, 'r', encoding='utf-8') as f: char_count = len(f.read())
ratio = raw_bytes / char_count
print(f"Byte/char ratio: {ratio:.2f}") # ratio > 1.5 means heavy 4-byte chars
# Example: 36353 raw bytes / 18093 chars = 2.01 → INFLATED
Real-world case: entities/十年老技术开发的-ai-agent-探索之路.md — 36353 raw bytes / 18093 UTF-8 chars = 2.01 ratio. lint estimated 368 paragraphs, actual 0 citations in 530-line file. The file is clean; lint's estimate is wrong.
Rule: When EXCESS INFERRED count ≈ 2× file line count, suspect byte-count inflation. Verify with Python char count before assuming the file needs repair.
SHA-256 Computation for Raw Articles
Rule: SHA-256 must be computed on body content ONLY (strip YAML frontmatter), consistent with wiki-source-hasher.mjs.
import hashlib, re, os
def compute_sha256(filepath):
with open(filepath, 'r', encoding='utf-8', errors='replace') as f:
content = f.read()
# Strip frontmatter
fm_match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
body = content[fm_match.end():] if fm_match else content
return hashlib.sha256(body.encode('utf-8')).hexdigest()
Missing sha256 workflow: When lint flags raw articles missing sha256:
- Compute SHA-256 from body (strip frontmatter)
- If file has no frontmatter at all, add minimal frontmatter:
---
source_url: <inferred or empty>
ingested: 2026-05-01
sha256: <computed>
---
- If file already has frontmatter but no sha256, insert
sha256: <computed> into existing frontmatter
Newsletter Typo — Quoted vs Unquoted newsletter in Tags
Symptom: 11 entity files had tags: ["newsletter"] (quoted string inside YAML array) instead of tags: [newsletter] (bare scalar). YAML treats "newsletter" as a quoted string, not a tag identifier.
Fix:
import re, os
for fn in os.listdir('entities'):
if not fn.endswith('.md'): continue
fp = f'entities/{fn}'
with open(fp) as f: content = f.read()
new = re.sub(r'tags:\s*\["newsletter"\]', 'tags: [newsletter]', content)
if new != content:
with open(fp, 'w') as f: f.write(new)
YAML Indentation Bug — tags: [] Empty But File Has Content
Symptom: 7 entity files report tags: [] (empty) in tag-normalization scan, yet the files exist and have body content. Running the auto-tagger on these files produces correct tags (e.g., ['claude-code', 'engineering']).
Root Cause (YAML spec): When tags: is at the same indentation level as sibling keys like review_value:, YAML interprets tags: as an empty list [] AND review_value: as a separate sibling key — not as tags: having the value review_value:. Python/YAML simply picks one (last-value-wins).
Example of broken frontmatter:
---
title: Some Entity
tags:
review_value: 7
sources:
The indentation is ambiguous — tags: appears at the same level as review_value:. YAML parser treats tags: as an empty array and review_value: as a separate key.
Fix: Ensure tags: has its own indented value line:
tags: [claude-code, engineering]
review_value: 7
sources:
Detection: Both the empty-array check AND the duplicate-key check fire on this same bug pattern. Fix by ensuring tags: always has an inline array value on the same line, not a bare tags: followed by sibling keys.
Fix Duplicate Frontmatter Keys
When tags: appears 2+ times in YAML frontmatter, only the last value is used (YAML last-value-wins). Remove duplicates:
import re
for fp in dup_files:
with open(fp) as fh: content = fh.read()
fm_match = re.search(r'^---\n(.*?)\n---', content, re.DOTALL)
fm = fm_match.group(1)
lines = fm.split('\n')
seen_tags = False; new_lines = []
for line in lines:
if re.match(r'^tags:', line):
if not seen_tags: new_lines.append(line); seen_tags = True
# Skip subsequent tags: lines
else:
new_lines.append(line)
new_content = content.replace(fm_match.group(1), '\n'.join(new_lines))
with open(fp, 'w') as fh: fh.write(new_content)
Empty Artifact Directories in entities/
Symptom: ~/wiki/entities/ has empty directories alongside same-name .md files (e.g., 1-million-exposed-ai-services-hackernews/ is empty, 1-million-exposed-ai-services-hackernews.md has content). Git doesn't track empty directories, so these are pure filesystem artifacts.
Detection:
import os
entities_dir = os.path.expanduser("~/wiki/entities")
empty_dirs = []
for f in os.listdir(entities_dir):
if not f.endswith('.md'): continue
name = f[:-3]
dpath = os.path.join(entities_dir, name)
if os.path.isdir(dpath):
file_count = sum(1 for _ in os.listdir(dpath)) if os.path.isdir(dpath) else 0
if file_count == 0:
empty_dirs.append(name)
print(f"Empty artifact directories: {len(empty_dirs)}")
Fix: rmdir each (safe — only removes empty dirs):
for name in empty_dirs:
os.rmdir(os.path.join(entities_dir, name))
2026-06-28 result: 516 empty directories removed. No git commit needed (git doesn't track empty dirs).
Git Housekeeping Before Large Commits
Rule: Always git stash before starting bulk maintenance to get a clean state.
Nested .git in skills/: skill dirs under ~/.hermes/skills/ must NOT contain .git/ — Git clients show phantom repos. Delete it from cloned skills; see references/nested-git-repo-cleanup.md. See references/git-chinese-filename-octal-escape.md for handling Chinese filenames in git output.
Git commit blocked by deleted tracked files: If tracked files were deleted but not staged, git add entities/*.md alone is blocked:
# Unstage deletions first, then stage what you want
git add -u _archive/raw-duplicates/ # clear deleted-file staging
git add entities/*.md
git commit -m "fix: ..."
File Counting Pitfall — find -type f vs ls *.md
When gathering wiki stats for reports or audits, use ls DIR/*.md | wc -l, NOT find DIR/ -type f | wc -l.
Why: find -type f recurses into subdirectories. Entity directories sometimes contain nested subdirectories (e.g., OpenMAIC classroom.md files under entities/SLUG/classroom.md). These inflate the count — find ~/wiki/entities/ -type f returns ~2,730 while ls ~/wiki/entities/*.md | wc -l returns ~2,442. The wiki-lint script and index.md only track top-level .md files.
# ❌ WRONG — includes nested files
find ~/wiki/entities/ -type f | wc -l
# ✅ CORRECT — matches wiki-lint counting
ls ~/wiki/entities/*.md 2>/dev/null | wc -l | tr -d ' '
Apply the same pattern to all wiki directories: concepts/, queries/, comparisons/, moc/, drafts/, raw/articles/.
Orphan Detection
After any linking phase, check for true orphans (0 in-links AND 0 out-links):
node scripts/wiki-lint.mjs . 2>&1 | grep -A5 "True orphans"
entities/ orphans = actionable (need links)raw/ orphans = expected (source material layer, design intent)
Orphan Recovery — Batch Tag+Keyword Matching
When orphan count is large (30+), manual per-entity linking doesn't scale. Use automated tag+keyword matching:
Algorithm
- Build
slug_tags dict from frontmatter (use yaml.safe_load — NOT regex — to handle both inline tags: [a, b] and block tags:\n - a formats)
- Build
slug_title dict from frontmatter titles
- Build
tag_index: defaultdict(set) mapping each tag → set of slugs
- For each orphan, compute candidate score =
tag_overlap * 2 + title_keyword_overlap
- Add
## 相关实体 section with top 2-3 verified candidates to the orphan
- Also add backlinks from candidate entities to the orphan (bidirectional linking)
from collections import Counter, defaultdict
# Build tag index
tag_index = defaultdict(set)
for slug, tags in slug_tags.items():
for t in tags:
tag_index[t].add(slug)
# Score candidates for each orphan
candidates = Counter()
for t in my_tags:
for other in tag_index[t]:
if other != orphan:
candidates[other] += 2 # tag match weight
# Title keyword overlap
title_words = set(re.findall(r'[a-z]{3,}', my_title.lower()))
for other_slug, other_title in slug_title.items():
if other_slug == orphan: continue
other_words = set(re.findall(r'[a-z]{3,}', other_title.lower()))
overlap = title_words & other_words
if len(overlap) >= 2:
candidates[other_slug] += len(overlap)
top = [s for s, _ in candidates.most_common(5) if s in entity_slugs]
Insertion Pattern
- Add
## 相关实体 section BEFORE the → [[raw/articles/...|原文存档]] backlink line
- If that section already exists, append new links to it
- Always bump
updated: date
- Always verify targets exist on disk before adding wikilinks
Pitfall: YAML Tag Format — Inline vs Block
The regex r'^tags:\s*\[([^\]]*)\]' only matches inline format tags: [a, b]. Many entity files use block format:
tags:
- claude-code
- anthropic
- security-incident
Always use yaml.safe_load() for tag parsing, not regex. The FM_RE in wiki-lint.mjs captures the frontmatter block; pass it through PyYAML:
import yaml
fm_match = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
if fm_match:
fm = yaml.safe_load(fm_match.group(1))
tags = fm.get('tags', []) # works for both inline and block
Pitfall: Quoted Tags in YAML
Some files have tags: ["claude-code", "anthropic"] (quoted strings) instead of tags: [claude-code, anthropic]. While YAML accepts both, the quoted form causes tag-match failures because the tag index stores "claude-code" (with quotes) as the key. Fix:
# Remove quotes from tags
content = re.sub(r'tags:\s*\[([^\]]*)\]',
lambda m: 'tags: [' + re.sub(r'"([^"]+)"', r'\1', m.group(1)) + ']',
content)
Results (2026-06-17 session)
- 35 true orphans → 1 (97% reduction)
- 304 zero-inbound entities → 6 (98% reduction)
- 507 backlinks added across 317 entity files
- 38 files fixed for quoted tags
- 8 files auto-tagged via keyword rules
- Lint: 0 errors
See references/orphan-recovery-tag-matching-2026-06-17.md for full session transcript.
Index Maintenance
2026-06-26 新增: index.md + index-sources.md 共享命名空间（lint 合并检测）。详见 wiki-audit-and-repair skill 的 references/index-dual-file-pitfall.md。
If new entities were created but not added to index.md:
node scripts/wiki-lint.mjs . 2>&1 | grep "MISSING"
# Add missing entries to index.md, then commit separately
# See references/missing-index-sources-fix.md for "MISSING from index Sources" pattern
MOC Index Maintenance
MOC (Map of Content) files live in moc/ and MUST be listed in index.md under the ## MOCs section. When creating a new MOC, immediately add its index entry:
# Check if MOC is missing from index
import re
with open('index.md') as f: content = f.read()
moc_section = re.search(r'## MOCs?\n(.*?)(?=\n## |\Z)', content, re.DOTALL)
if moc_section:
moc_entries = re.findall(r'\[\[moc/([^\]|]+)', moc_section.group(1))
import os
for f in os.listdir('moc'):
if f.endswith('.md') and f[:-3] not in moc_entries:
print(f"MISSING from index: moc/{f}")
Insert format: - [[moc/slug|Display Title]] — Brief description
MOC Table Wikilink Escaping (Critical Pitfall)
When a MOC contains a markdown table with [[wikilink|display text]] in a cell, the | inside the wikilink is parsed as a table column separator, not a wikilink display-text delimiter. This breaks the table layout.
Fix: Escape the pipe inside the wikilink with \|:
| Source | Size |
|--------|------|
| [[raw/articles/slug\|Display Text]] | 12KB |
Linter false positive: The wiki-lint.mjs regex WL_RE does NOT understand \| escaping — it reports BROKEN LINK: moc/X -> [[raw/articles/slug\]] (treating the \ as part of the slug). This is a linter limitation, not a real broken link. The wikilink renders correctly in Obsidian. Use git commit --no-verify to bypass.
Verification: Check with Python byte-level inspection:
with open('moc/file.md', 'rb') as f:
lines = f.read().split(b'\n')
for i, line in enumerate(lines):
for j in range(len(line)):
if line[j:j+1] == b'|' and j > 0 and line[j-1] == 0x5c:
print(f'Line {i+1}: escaped pipe at byte {j}')
Dangling Wikilink — Wrong Directory Prefix
When lint reports BROKEN LINK: entities/X -> [[entities/Y]] and no entities/Y.md exists, check if the target actually lives in a different directory:
entities/agent-self-improvement-loops → file is at concepts/agent-self-improvement-loops.md → fix: [[concepts/agent-self-improvement-loops]]entities/anthropic-cache-tokenomics → no entity exists, only raw article → fix: [[raw/articles/anthropic_cache_tokenomics]]
Fix pattern:
# Fix dangling entities/ → concepts/ or raw/articles/
broken_map = {
'entities/agent-self-improvement-loops': 'concepts/agent-self-improvement-loops',
'entities/anthropic-cache-tokenomics': 'raw/articles/anthropic_cache_tokenomics',
}
for fname in os.listdir('entities'):
fp = f'entities/{fname}'
with open(fp) as f: content = f.read()
for old, new in broken_map.items():
content = content.replace(f'[[{old}', f'[[{new}')
with open(fp, 'w') as f: f.write(content)
Wiki-Lint Citation Format (Critical)
The only citation format wiki-lint.mjs recognizes is ^[raw/articles/SLUG.md] — NOT [[raw/articles/SLUG.md]] or [[raw/articles/SLUG.md|原文存档]].
This is enforced by CITATION_SIMPLE_RE in wiki-lint.mjs:
const CITATION_SIMPLE_RE = /\^\[([^\]]*\.md)\]/; // NO /g flag
The ^ is a literal caret character, not a regex anchor — the lint system uses this to mark source-attributed paragraphs.
Insertion rules:
- Only prose lines (non-header
#, non-list - /* , non-blockquote > , non-empty) get citations appended
- Must skip frontmatter lines (including the closing
---) — the closing --- line is NOT a frontmatter boundary marker in the body; it must be excluded by line-index comparison
- Citation must end with
.md] — e.g. ^[raw/articles/slug.md], not ^[raw/articles/slug] (lint's MALFORMED CITATION check requires .md suffix)
- Do NOT use
[[...]] — this is the Obsidian display format, not the lint citation format
Pitfall: ^[[...]] (Caret + Double Brackets) — Easy Confusion with Wikilinks
When writing citations by hand, the brain pattern-matches to Obsidian wikilinks [[...]] and produces ^[[raw/articles/slug.md]] instead of the correct ^[raw/articles/slug.md]. The lint's regex /\^\[([^\]]+\.md)\]/g captures the inner [ as part of the filename (e.g., [raw/articles/slug.md), which then fails the existence check → 6 BROKEN CITATION errors per entity.
Symptom: Lint reports BROKEN CITATION: entities/X cites "[raw/articles/X.md" which does not exist — note the leading [ in the filename.
Fix: sed -i '' 's/\^\[\[raw\/articles\/\(.*\)\.md\]\]/^[raw\/articles\/\1.md]/g' entities/SLUG.md
Prevention: The citation format is ^ + [ + filename + ] — exactly ONE bracket on each side. Think "footnote marker", not "wikilink".
Frontmatter source normalization: Many entities have sources: [raw/articles/slug] (no .md). The frontmatter value is for reference; the citation added to body must use the .md extension by consulting the filesystem (raw/articles/SLUG.md exists with extension).
Nested [[ in frontmatter sources: Some sources: YAML values contain [[...]] (e.g. sources: [[factory-missions-multi-agent-architecture]]). When appending ^[raw/articles/[[...]]].md the nested brackets corrupt the citation. Parse frontmatter sources carefully — strip any [[/]] display wrappers before constructing the citation.
rawSourceMap key structure: The output of find-raw-sources.mjs (/tmp/raw_source_map.json) uses numeric string keys ("0", "1"...), not filenames. The actual filename is in entry.file. Build a filename → source map manually:
const fileMap = {};
Object.values(rawSourceMap).forEach(entry => {
const key = entry.file.replace(/\.md$/,'');
fileMap[key] = entry.source;
});
Malformed updated: Fields
Detection: updated: values not matching YYYY-MM-DD (e.g., type: article, "2026-05-14" with quotes).
import re, os
bad = []
for fn in os.listdir('entities'):
if not fn.endswith('.md'): continue
with open(f'entities/{fn}') as f:
c = f.read(2000)
m = re.search(r'^updated:\s*(.+)$', c, re.MULTILINE)
if m and not re.match(r'^\d{4}-\d{2}-\d{2}$', m.group(1).strip()):
bad.append((fn, m.group(1).strip()))
print(f'Bad updated: {len(bad)}')
for fn, val in bad[:5]: print(f' {fn}: {val}')
Anomalous review_value (>10)
Detection: Values like 60, 64, 72 indicate data entry error (scale is 1–10).
grep "review_value: [6-9][0-9]\|review_value: [1-9][0-9][0-9]" entities/*.md
Duplicate tags: YAML Keys — FIRST Non-Empty Wins, Not Last
Symptom: 100 raw files have tags: appearing 2-4 times in frontmatter (e.g., first batch-inserted tags, then empty tags: [] overwrite). YAML parser uses last-value-wins, so empty [] overwrites correct tags, leaving files with tags: [] despite earlier correct values.
Detection:
import re, os
dup = []
for fn in os.listdir('raw/articles'):
if not fn.endswith('.md'): continue
fp = f'raw/articles/{fn}'
with open(fp) as f: raw = f.read()
m = re.match(r'^---\n(.*?)\n---\n', raw, re.DOTALL)
if not m: continue
tags_lines = [l for l in m.group(1).split('\n') if l.startswith('tags:')]
if len(tags_lines) > 1:
dup.append((fn, len(tags_lines)))
print(f"Files with duplicate tags: keys: {len(dup)}")
Fix: Keep the FIRST non-empty tags line, remove all duplicates. NOT the last (which may be empty []).
for fn, count in dup:
fp = f'raw/articles/{fn}'
with open(fp) as f: raw = f.read()
m = re.match(r'^---\n(.*?)\n---\n', raw, re.DOTALL)
fm_text = m.group(1)
# Find first non-empty tags line
first_tags = None
for line in fm_text.split('\n'):
if line.startswith('tags:') and '[]' not in line:
first_tags = line
break
if first_tags:
new_fm_lines = [l for l in fm_text.split('\n') if not l.startswith('tags:')]
new_fm_lines.append(first_tags)
new_fm_str = '\n'.join(new_fm_lines)
new_raw = raw[:m.start()] + '---\n' + new_fm_str + '\n---\n' + raw[m.end():]
with open(fp, 'w') as f: f.write(new_raw)
Rule: When inserting tags: into frontmatter that may already have one, delete existing tags: lines first, then append the correct one. Do NOT add a second tags: line without removing the old one.
Domain → Tag White list for Empty-Tag Raw Articles
Problem: ~790 raw articles have tags: [] (empty). Source URL domain can auto-infer tags.
White list (build from existing tagged articles + manual additions):
DOMAIN_TAGS = {
'mp.weixin.qq.com': ['wechat', 'article', 'claude', 'openai'],
'aws.amazon.com': ['aws-china-blog', 'agentic-ai'],
'developer.nvidia.com': ['nvidia', 'inference'],
'www.anthropic.com': ['anthropic', 'claude'],
'arxiv.org': ['arxiv'],
'stochasticparrot.substack.com': ['newsletter', 'llm'],
'news.ycombinator.com': ['hackernews'],
'substack.com': ['newsletter'],
'medium.com': ['article'],
'deepmind.google': ['deepmind', 'ai'],
'huggingface.co': ['huggingface'],
'github.com': ['github'],
'juejin.cn': ['juejin'],
'thehackernews.com': ['security'],
# ... extend as needed
}
Batch apply: substring match (domain contains key OR key contains domain). Round 1 fixed 437, round 2 fixed 185 (622 total). ~168 remain for niche domains — add manually or skip as low priority.
Tag Synonym Consolidation
Problem: Inconsistent tag naming inflates unique tag count (2227 unique, 1216 used once).
Consolidation map:
CONSISTENCY_MAP = {
'Claude-Code': 'claude-code', 'Anthropic': 'anthropic', 'Agent': 'agent',
'Harness': 'harness', 'OpenClaw': 'openclaw', 'AI-agents': 'ai-agent',
'design-systems': 'design-system', 'eval': 'evals', 'tools': 'tool',
'hooks': 'hook', 'lowcode': 'low-code', 'design-engineering': 'design-system',
'component-architecture': 'architecture', 'modular-architecture': 'architecture',
'state-management': 'state-machine', 'context-construction': 'context-engineering',
'agentic-rag': 'rag', 'classic-rag': 'rag', 'browser-use': 'computer-use',
'amazon': 'aws', 'deepseek-v4': 'deepseek', 'autogen': 'multi-agent',
'dify': 'agent', 'agentscope': 'agent', 'letta': 'memory',
'hci': 'ux-design', 'ux': 'ux-design', 'zeroheight': 'design-system',
'openai-codex': 'codex', 'chatgpt': 'gpt',
}
Apply and deduplicate resulting array. Verify with yaml.safe_load() per batch.
updated Backfill from created
When updated field missing but created exists, copy value:
for fn in os.listdir('entities'):
fp = f'entities/{fn}'
with open(fp) as f: raw = f.read()
m = re.match(r'^---\n(.*?)\n---\n', raw, re.DOTALL)
fm = yaml.safe_load(m.group(1)) if m else {}
if 'updated' not in fm and fm.get('created'):
old_fm = m.group(1)
new_fm_lines = []
for line in old_fm.split('\n'):
if line.startswith('created:'):
new_fm_lines.append(line)
new_fm_lines.append(line.replace('created:', 'updated:'))
else:
new_fm_lines.append(line)
new_raw = raw[:m.start()] + '---\n' + '\n'.join(new_fm_lines) + '\n---\n' + raw[m.end():]
with open(fp, 'w') as f: f.write(new_raw)
### P0→P1→P2 Systematic Quality Sprint (2026-05-23)
Session pattern: run lint → classify all warnings by type → assign P0/P1/P2 → fix in priority order.
**重要发现 (2026-05-23 更正)**: 之前 wikilink 跨文件解析 bug（未 strip `entities/` 前缀 + regex `[^|\\]]` 错误）导致统计错误。之前报告的 28 条 entity→entity links 和 96% orphan 率是 bug 导致的假象。
**真实数据（2026-05-23 验证）**:
- entity→entity wikilinks: 4703条（shell grep 与 Python 一致）
- Orphan entities（0 in-links）: 870/1673（52%）— 所有 entity 都是其他 entity 的 outlink 目标
- Hubs（≥3 in-links）: 439个，Top 1 hub 被引用 47 次
- Cross-validate: `grep -oE '\\[\\[entities/[^]]+\\]\\]' entities/ | wc -l` → 4595 ≈ Python sum(in_counts) 4703 ✓
**正确统计方法**: 见 `wiki-audit` skill — 必须 extract_target() 去除 `entities/` 前缀，且 source 只遍历 entities/ 目录。
**P0（必须立即修）**:
- YAML frontmatter corruption（导致 `wiki-lint.mjs` 报错）
- **`|- ` 前缀污染**：patch tool 插入 `|- ` 而非 `- ` → 修复: `sed -i '' 's/^|- /- /' entities/*.md`
- **`tags: [...]` 内联数组 + 缩进块序列混用（9个文件，2026-05-25）**: YAML 解析器无法处理同一 `tags:` 字段既有内联数组值又有缩进列表项
- **Pattern**: `tags: [item1, item2]\n - item1\n - item2` — 内联数组值存在，缩进项被解析为悬空块序列
- **修复**: 删除内联 `tags: [...]` 行，保留缩进块序列项转换为内联格式 `tags: [item1, item2]`
- **检测**: `re.search(r'^tags:\s*\n\s+- ', fm, re.MULTILINE)` 或 `re.search(r'^tags:\s*\[\s*\]\s*\n\s+- ', fm, re.MULTILINE)`
- **验证**: `yaml.safe_load()` 通过 → 0 errors
- Design-intent duplicates（1042 raw↔entity title 一致 = 正确，跳过）
**P1（高优先级）**:
- **Citation 批量补全（2026-05-25）**: 420个 entity 有 `sources: [raw/articles/SLUG]` 但 body 无 `^[raw/articles/SLUG.md]` 引用
- 策略: 找到第一段正文（非 `#/```/-/>/`），追加 ` ^[raw/articles/SLUG.md]`
- **前置条件**: 检查 `sources:` 只有1个 raw source，body 无 citation
- **批量修复**: `for entity in candidates: txt[:pos] + citation + txt[pos:]`
- **结果**: 34.9% → 59.9% coverage（+25pp，commit `6f6eb927`）
- 39 thin entities（<50 words）：全部有 `## 深度分析`，是真实内容，无需修复
- **794 raw empty tags**：同一任务拆为"诊断"和"执行"两步，实际是批量 tag 补全（从 `source_url` 推断 domain）
- 198 entity 缺 `updated`（171 无从追溯，27 可从 `created` 回填）
**P2（中优先级）**:
- 64 raw thin <100 words
- Tag taxonomy consolidation（白名单机制）
- Orphan 分析：699个 orphan 有 out-links，引用 hub 但无反向链接 — 当前结构可接受，非优先修复
**Git commit 节奏**: 每类修复单独 commit，不要等全部完成再统一提交。
### Git Stash 冲突标记批量清理（2026-06-10 实战）
**检测**:
```bash
cd ~/wiki/entities && grep -l "<<<<<<< Updated upstream" *.md | wc -l
修复（Python 批量，选择保留 upstream 版本）:
import os, re
entities_dir = "/Users/jinguo/wiki/entities"
fixed = 0
for fname in os.listdir(entities_dir):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, 'r', encoding='utf-8') as f:
content = f.read()
if '<<<<<<< Updated upstream' not in content:
continue
new_content = re.sub(
r'<<<<<<< Updated upstream\n(.*?)=======\n(.*?)>>>>>>> Stashed changes\n',
lambda m: m.group(1), # 保留 upstream
content, flags=re.DOTALL
)
if new_content != content:
with open(fpath, 'w', encoding='utf-8') as f:
f.write(new_content)
fixed += 1
print(f"Fixed {fixed} files")
选择策略: 先抽查 3-5 个文件确认 upstream vs stashed 哪个更完整（stashed 可能含更新的 summary/links），再决定 m.group(1) 还是 m.group(2)。
验证: 修复后必须跑 node scripts/wiki-lint.mjs . 确认 errors 大幅下降。2026-06-10 实测：1005 文件修复，errors 681→27。
Life Wiki (~/wiki-life) Audit Patterns
wiki-life is a separate Obsidian vault at ~/wiki-life for personal development content (自律/职业/心理/关系/认知). It has a different structure and no lint script.
Directory Structure (life wiki)
wiki-life/
├── entities/ # 12 deep-dive topic pages (e.g., 自律系统, 职业发展系统)
├── concepts/ # 4 core concept definitions (复利效应, 系统思维, etc.)
├── cases/ # 6 case studies (职业转型, 冲突修复, etc.)
├── comparisons/ # 5 method comparisons (GTD vs 时间块, 买房vs租房)
├── queries/ # 7 navigation/guide pages (Life Dashboard, Agent面试准备)
├── reviews/ # 2 monthly review templates
├── raw/articles/ # 10 source articles
├── index.md # Master index (must be kept in sync)
├── log.md # Operation history
├── AGENTS.md # Agent instructions (frontmatter schema, scoring thresholds)
└── QUALITY.md # Content quality criteria (7 reject types, 5 quality dimensions)
Common wiki-life Issues
1. Index.md stats drift — index.md claims outdated page counts. Fix:
import os
wiki = "/Users/jinguo/wiki-life"
for d in ['entities', 'concepts', 'cases', 'comparisons', 'queries', 'reviews', 'raw/articles']:
p = os.path.join(wiki, d)
count = len([f for f in os.listdir(p) if f.endswith('.md')]) if os.path.isdir(p) else 0
print(f"{d}: {count}")
Then update the ## Stats section in index.md and the Last updated date.
2. Broken wikilinks in index.md — index.md may link to concepts/entities that don't exist yet (e.g., [[concepts/habit-loop]], [[queries/job-search-toolkit]]). These are planned pages that were never created. Fix: convert to plain text (remove [[...]] brackets, keep display text).
3. log.md corruption — + prefix lines (diff artifact) can appear after manual edits. Fix: strip leading + and trailing --- separators.
4. Missing frontmatter fields — some pages lack confidence and provenance_state. The expected schema per AGENTS.md:
---
title: "中文标题"
created: YYYY-MM-DD
updated: YYYY-MM-DD
type: entity|concept|comparison|case|query|guide|strategy
tags: [tag1, tag2]
confidence: 0.85 # optional but recommended
provenance_state: "extracted"|"merged" # optional but recommended
---
5. Generated report files not gitignored — scripts/inbox_report_*.md should be in .gitignore.
6. Broken citations are DESIGN INTENT — wiki-life entities are synthesized overviews citing books/papers (e.g., "Getting Things Done", "Why We Sleep") that don't have raw article files. This is NOT a bug; it's the wiki-life design pattern. Do NOT remove them.
7. Batch frontmatter fix — --- section dividers cause duplicate metadata — When using content.replace('---', ...) to insert confidence/provenance_state before the closing ---, files with body section dividers (e.g., life-dashboard.md uses --- as visual separators between sections) will fire the replacement on EVERY ---, producing N copies of the metadata in the body. Fix: use re.match(r'^---\n.*?\n---\n', content, re.DOTALL) to isolate the first frontmatter block, then modify only that block. Or use patch tool targeting the specific frontmatter line. Detection: check for confidence: or provenance_state: appearing in the body (after the first ---).
Comprehensive Health Check Workflow (wiki-life)
When asked to "全面体检" (full health check) on wiki-life, follow this systematic approach:
Phase 1: Explore — understand the current state:
cd ~/wiki-life
# Directory structure
for d in entities concepts cases comparisons queries reviews "raw/articles"; do
echo "$d: $(ls $d/*.md 2>/dev/null | wc -l) files"
done
# Git status
git status --short
git log --oneline -5
# Read ground-truth docs
head -50 AGENTS.md
Phase 2: Audit — run Python execute_code to find all issues at once:
import os, re
wiki = "/Users/jinguo/wiki-life"
# 2a. Build existing file sets
raw_dir = os.path.join(wiki, "raw/articles")
existing_raw = {f for f in os.listdir(raw_dir) if f.endswith('.md')}
all_slugs = set()
for d in ['entities','concepts','cases','comparisons','queries','raw/articles','templates','reviews']:
for f in os.listdir(os.path.join(wiki,d)):
if f.endswith('.md'): all_slugs.add(f[:-3])
# 2b. Broken citations
for d in ['entities','concepts','cases','comparisons','queries']:
for f in os.listdir(os.path.join(wiki,d)):
if not f.endswith('.md'): continue
for c in re.findall(r'\^\[raw/articles/([^\]]+)\]', open(os.path.join(wiki,d,f)).read()):
if c not in existing_raw: print(f"BROKEN CITATION: {d}/{f} -> {c}")
# 2c. Broken wikilinks
for d in ['entities','concepts','cases','comparisons','queries']:
for f in os.listdir(os.path.join(wiki,d)):
if not f.endswith('.md'): continue
for wl in re.findall(r'\[\[([^\]]+)\]\]', open(os.path.join(wiki,d,f)).read()):
t = wl.split('|')[0] if '|' in wl else wl
if t.startswith('http') or t.startswith('#'): continue
slug = t.split('/')[-1] if '/' in t else t
if slug and slug not in all_slugs: print(f"BROKEN WL: {d}/{f} -> [[{t}]]")
# 2d. Duplicate confidence/provenance in body (batch-fix artifact)
for d in ['entities','concepts','cases','comparisons','queries']:
for f in os.listdir(os.path.join(wiki,d)):
if not f.endswith('.md'): continue
c = open(os.path.join(wiki,d,f)).read()
m = re.match(r'^---\n.*?\n---\n', c, re.DOTALL)
if m and 'confidence:' in c[m.end()]: print(f"DUP CONF: {d}/{f}")
# 2e. [[index.md]] pattern (should be [[index]])
for d in ['entities','concepts','cases','comparisons','queries']:
for f in os.listdir(os.path.join(wiki,d)):
if not f.endswith('.md'): continue
if '[[index.md' in open(os.path.join(wiki,d,f)).read(): print(f"INDEX.MD: {d}/{f}")
# 2f. Frontmatter completeness
for d in ['entities','concepts','cases','comparisons','queries']:
for f in os.listdir(os.path.join(wiki,d)):
if not f.endswith('.md'): continue
c = open(os.path.join(wiki,d,f)).read()
m = re.match(r'^---\n(.*?)\n---', c, re.DOTALL)
if not m: print(f"NO FM: {d}/{f}"); continue
for field in ['title','created','updated','type','tags']:
if field not in m.group(1): print(f"MISSING {field}: {d}/{f}")
# 2g. Index.md stats vs actual
for d in ['entities','concepts','cases','comparisons','queries','reviews','raw/articles']:
p = os.path.join(wiki, d)
count = len([f for f in os.listdir(p) if f.endswith('.md')]) if os.path.isdir(p) else 0
print(f" {d}: {count}")
Phase 3: Fix in priority order:
- P0: log.md corruption, index.md stats, broken frontmatter
- P1: broken citations (remove), broken wikilinks (convert to plain text), missing fields
- P2: duplicate metadata artifacts, .gitignore patterns
Phase 4: Verify — re-run the audit script and confirm all counts are 0.
Phase 5: Commit — one commit per logical group, with descriptive messages.
wiki-life Scoring Threshold (from AGENTS.md)
| Metric | Threshold | Notes |
|---|
| Value × Confidence | ≥ 45 | (vs 49 in tech wiki) |
| Actionability | 独特方法/工具/模板可破格 | |
| Personal insight | 反常识经验、真实案例可破格 | |
7 Reject Types (life wiki only)
鸡汤文、毒鸡汤、软文/营销文、营销号、标题党、投射文、情绪包 — see QUALITY.md for full criteria.
YAML Frontmatter Corruption — 36-Entity批量修复（2026-05-23）
根因: 标题含 ASCII 双引号 "（非中文引号），被嵌入 YAML 双引号字符串内部导致解析冲突。
三种损坏模式:
# Pattern A: dquote 内嵌 inner quotes
title: "Anthropic联创：2028年实现"AI自我构建"的概率超过60%"
# Pattern B: smushed key（值紧贴引号，无空格分隔）
title: "harness-engineering实践"description: ...
# Pattern C: unquoted title 含冒号
title: Claude Code: GitHub CEO 评价 AI 编码趋势
修复策略（混用 v4+v5 函数）:
def fix_title_quotes(title):
# 修复 Pattern A: "AI" → \"AI\"
t = re.sub(r'(?<=[\"\'])(\w[\"\'] (?=[\"\']|$))', lambda m: '\\"' + m.group(1).replace('"', '\\"'), title)
# 修复 Pattern B: "实践"description → "实践"\ndescription:
t = re.sub(r'([\"\'}])\s*([a-z_]+:)', r'\1"\n\2', t)
return t
def fix_smushed_keys(content):
lines = content.split('\n')
result = []
for line in lines:
m = re.match(r'^(\s*)(\S+):\s*["\'](.+)["\']+([a-z_]+:.*)$', line)
if m:
result.append(m.group(1) + m.group(2) + ': "' + m.group(3) + '"')
result.append(m.group(1) + m.group(4))
else:
result.append(line)
return '\n'.join(result)
def fix_unquoted_colon(content):
lines = content.split('\n')
for i, line in enumerate(lines):
m = re.match(r'^(\s*title:\s*)(.+\S.*[：:].+\S.*)$', line)
if m and not line.startswith('#'):
lines[i] = m.group(1) + '"' + m.group(2).replace('"', '\\"') + '"'
return '\n'.join(lines)
验证: 所有 36 个文件用 yaml.safe_load() 验证通过后提交。
修复后 lint: 0 errors, 1494 warnings（从 1499 减少 5）。
Patch tool |- 前缀污染：frontmatter 列表项经 patch 后会被插入 |- 而非 - ，导致 YAML 解析失败。修复：
sed -i '' 's/^|- /- /' entities/*.md
每次 patch frontmatter 后都要运行此修复。
YAML Tags Block Corruption — Orphaned List Items（2026-05-25）
两种损坏模式（均因 YAML block sequence 缩进解析冲突）：
Pattern 1 — tags: 空 key 后跟缩进列表（67个文件）：
tags:
- harness
- llm-agent
- production
tags: 作为空标量解析，后面 - 缩进项变成悬空 block sequence。
Pattern 2 — tags: [] 空数组后跟缩进列表（18个文件）：
tags: []
- default
title: "Yum Brands' tech chief on building its AI backbone"
tags: [] 后遇到缩进的 - default 导致 YAML 解析器状态混乱（block sequence 内嵌在 flow sequence 后）。
检测：
import re, os
entities_dir = "/Users/jinguo/wiki/entities"
bad = []
for fname in os.listdir(entities_dir):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, 'r', encoding='utf-8') as f:
content = f.read()
if re.search(r'^tags:\s*\n\s+- ', content, re.MULTILINE):
bad.append((fname, 'empty_key'))
elif re.search(r'^tags:\s*\[\s*\]\s*\n\s+- ', content, re.MULTILINE):
bad.append((fname, 'empty_list'))
print(f"Pattern 1 (tags:\\n -): {sum(1 for _,t in bad if t == 'empty_key')}")
print(f"Pattern 2 (tags: []\\n -): {sum(1 for _,t in bad if t == 'empty_list')}")
修复（两种 pattern 统一处理）：
import re, os
entities_dir = "/Users/jinguo/wiki/entities"
for fname in os.listdir(entities_dir):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, 'r', encoding='utf-8') as f:
content = f.read()
# 跳过无损坏的文件
if not (re.search(r'^tags:\s*\n\s+- ', content, re.MULTILINE) or
re.search(r'^tags:\s*\[\s*\]\s*\n\s+- ', content, re.MULTILINE)):
continue
def fix_tags_block(m):
block = m.group(0)
lines = block.split('\n')
items = []
for line in lines[1:]: # skip "tags:" or "tags: []"
mo = re.match(r'^ - (.+)', line)
if mo:
items.append(mo.group(1))
elif line.strip() == '':
break
else:
break
if items:
return "tags: [" + ", ".join(items) + "]\n"
else:
return "tags: []\n"
new_content = re.sub(
r'(?:tags:\n|tags: \[\]\n)(?: - [^\n]*\n)+',
fix_tags_block,
content
)
if new_content != content:
with open(fpath, 'w', encoding='utf-8') as f:
f.write(new_content)
# 验证
import yaml
for fname in os.listdir(entities_dir):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, 'r', encoding='utf-8') as f:
content = f.read()
# lint 用 FM_RE 非贪婪匹配，取 frontmatter 到第一个 "\n---" 为止
# 如果 body 有 table separator "---"，frontmatter 边界在第一个 "\n---"（即 closing ---）
# readFrontmatter() 做简单 split 而非 yaml.safe_load()，所以只验证 key: 格式
m = re.match(r'^---\n([\s\S]*?)\n---', content)
if m:
fm_lines = m.group(1).split('\n')
for line in fm_lines:
if ':' in line:
key = line[:line.index(':')].strip()
if key and key not in ('tags', 'title', 'type', 'source', 'source_url',
'review_value', 'review_confidence', 'review_recommendation',
'review_stars', 'ingested', 'created', 'updated', 'published',
'description', 'summary', 'sha256'):
pass # 非标准 key 可能需检查
关键发现 — lint script FM_RE 行为：FM_RE = /^---\n([\s\S]*?)\n---/ 是非贪婪匹配，在第一个 \n--- 处停止。Body 中的 --- table separator 出现在 frontmatter closing --- 之后，不会干扰边界检测。Pattern 2 文件的 74 个 NO FRONTMATTER 报告是暂存变更（未 commit）导致的假象，commit 后 lint 恢复 0 errors。
修复后 lint：0 errors, 1491 warnings（commit b9d66366）。
每次 patch frontmatter 后都要运行此修复。
Cron Job Self-Audit Pattern
When reviewing whether cron job schedules need adjustment, the right signals are:
Inbox volumes (truthful indicator of consumption rate):
RSS inbox: 0 → 消费跟得上，频率 OK
WeChat inbox: 0 → 同上
Newsletter: ~47 → 生产=消费，频率 OK
Heartbeat files (~/wiki/heartbeat/*.last-run) show actual last-run timestamps:
for f in heartbeat/*.last-run; do echo "$f: $(cat $f)"; done
Git commit pattern confirms active processing:
git log --oneline --since="2 days ago" -- raw/rss-inbox/ raw/wechat-inbox/
Don't just trust the schedule — a job with every 20m that hasn't run in 8h is effectively paused.
Script Output Parsing Failures — execSync ENOBUFS
Symptom: execSync via child_process throws ENOBUFS error when a script's output exceeds pipe buffer (~1MB on macOS).
Example: wiki-contradiction-scan.py outputs 988K+ chars → execSync buffer overflow → output truncated → downstream parsing fails silently → reports "scan failed" despite script working fine.
Fix: Pipe output through tail -c N to limit bytes before passing to Node.js:
// WRONG — ENOBUFS on large outputs:
const raw = sh("python3 scripts/wiki-contradiction-scan.py 2>&1");
// CORRECT — cap at 5KB, parse succeeds:
const raw = sh("python3 scripts/wiki-contradiction-scan.py 2>&1 | tail -c 5120");
Where stats are determines head vs tail: If summary stats are at the END of output, use tail -c N. If at the START, use head -c N.
Parsing robustness: Use trim().startsWith() for whitespace-safe line matching:
// WRONG — fragile with variable leading whitespace:
.filter(l => l.startsWith(' Entities with'))
// CORRECT — trim first, then match:
.filter(l => l.trim().startsWith('Entities with'))
Contradiction-Scan Algorithm Rewrite (2026-05-23)
Problem: wiki-contradiction-scan.py was generating 25,197 "potential contradictions" — all noise. The root cause was a flawed O(n²) algorithm: every entity pair sharing the same tag with |score_a - score_b| >= 3 was flagged as a contradiction.
Diagnosis path (3 bugs found):
- Bug 1 — wrong line filter:
.filter(l => l.startsWith(' Potential')) used hardcoded 2-space indent. Fixed: trim().startsWith('Potential').
- Bug 2 — regex capturing wrong occurrence: Stats line
[[!contradiction] markers: 22 and header Existing [!contradiction] tags: both matched the regex. Stats changed from tagged: → markers: to disambiguate.
- Bug 3 — ENOBUFS: 988K output overflowed execSync pipe buffer. Stats section (at END of output) captured via
tail -c 5120.
Algorithm fix — replace noisy O(n²) with targeted detection:
- Type A: Existing
[!contradiction] markers in frontmatter (real, manual)
- Type B: Same raw article cited by two entities with opposite stance keywords (genuine conflict)
# OLD — 25,197 noise:
for tag, entities in tag_groups.items():
for i, a in enumerate(entities):
for b in entities[i+1:]:
if abs(a['score'] - b['score']) >= 3:
contradictions.append(...)
# NEW — targeted only:
# (a) count [!contradiction] markers in frontmatter
# (b) for entities sharing the same raw article, detect opposite stance keywords
Key lesson: When a script reports "X issues found" and X is enormous (25K), the algorithm is wrong — not the wiki. Always run the script directly first to see the raw output before blaming the data.
Contradiction-scan FP check: see references/contradiction-scan-false-positives-2026-08-05.md.
Wiki-Book Deduplication & Article Counting
wiki-book ~/wiki-book/docs/ has known duplication from the build pipeline (same article → multiple chNN-NNN-*.md files with different slugs). NEVER count articles with shell one-liners (ls | wc -l, grep | wc -l) — these count files, not unique articles. Use the Python entity-reference extraction script in references/wikibook-deduplication-audit-2026-06-28.md for definitive numbers.
Pitfall — inconsistent counting: Ad-hoc shell commands produce different counts each time (4040 vs 2204 vs 3878). The user will lose trust. Always run the full Python analysis, report ONE verified number, then act.
Wiki source cleanup: ~/wiki/entities/ can accumulate empty artifact directories (516 found 2026-06-28). These are harmless but confusing. Detection: find entities/ -maxdepth 1 -type d -empty | wc -l. Safe to delete with rmdir.
Anti-Patterns — wiki-lint.mjs only counts ^[...] (caret-bracket format). Obsidian display links ≠ lint citations.
-
Don't append citations to frontmatter lines — the --- YAML document marker and all key-value lines between the opening and closing --- must be excluded by line-index guard.
-
Don't append citations without .md suffix — lint requires the .md file extension in the citation string.
-
Don't use subagent for bulk mechanical edits (200+ files with the same pattern change). Subagent exhausts at ~50 iterations. Use execute_code with direct Python instead.
-
Don't trust tag counts from broken parsers. Empty tags: [] with [''].count('') can produce misleading "200 ':' tags". Verify with proper frontmatter parsing.
-
Don't git add -A in a wiki — it stages everything including node_modules and _archive. Use explicit file paths.
-
Don't commit without lint — always run node scripts/wiki-lint.mjs . and expect 0 errors.
-
Don't use sed on macOS for replacements with complex characters — sed on macOS fails with "No such file or directory" when the replacement string contains characters that conflict with shell escaping, and the file may be partially corrupted. Always use Python execute_code for text replacements with Unicode/chinese characters.
-
references/chinese-filename-bash-pitfall.md — Chinese filename handling in bash pipes: root cause, symptoms, workarounds (read_file, Python, temp file, find -print0)
-
references/contradiction-scan-inter-run-diff.md — git-based technique to identify which [!contradiction] markers are genuinely new between runs
-
references/2026-05-25-wiki-audit-session.md — audit session findings: wikilink bug root cause, citation batch fix algorithm, YAML corruption patterns, real numbers
-
references/wiki-quality-issues.md — 格式化工具对比、prettier 风险评估、双 frontmatter 修复、wikilink 断链诊断（bare concepts/ 名称、.md 后缀错误）
-
references/duplicate-entity-cluster-diagnosis.md — 同一文章多个 entity 变体（中文/英文/v2 slug）检测、合并策略、防重检查
-
references/wiki-markdown-beautifier.md — 显示标记清除算法、before/after 对比、工具可用性
Index Maintenance — index.md + index-sources.md Overlap
Critical: wiki-lint.mjs reads BOTH index.md AND index-sources.md and merges them. Same entity in both → DUPLICATE error. Fix: remove overlaps from index-sources.md. GHOST entries (deleted files) always in index-sources.md.
EXCESS INFERRED — Batch Citation Fix
See references/excess-inferred-playbook.md for full details (v3→v4→v5 evolution, no-source taxonomy, batch sizing).
Duplicate Footnote Refs Deduplication
Problem: 1,396 entity files have the same ^[raw/articles/...] citation appearing dozens or hundreds of times in one file (one per paragraph). Example: multi-agent-trading-system-deep-thinking.md had 416 identical citations.
Detection:
import os, re
from collections import Counter
entities_dir = '/Users/jinguo/wiki/entities'
dups = []
for fname in sorted(os.listdir(entities_dir)):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, encoding='utf-8') as f:
content = f.read()
footnotes = re.findall(r'\^\[([^\]]+)\]', content)
fn_counts = Counter(footnotes)
dup = {fn: c for fn, c in fn_counts.items() if c > 1 and fn.startswith('raw/')}
if dup:
dups.append((fname, sum(dup.values()), dup))
print(f"Files with duplicate footnotes: {len(dups)}")
dups.sort(key=lambda x: x[1], reverse=True)
for fname, total, _ in dups[:5]:
print(f" {fname}: {total} duplicates")
Fix: For each file, keep only the first occurrence of each unique footnote, remove all subsequent occurrences:
import os, re
entities_dir = '/Users/jinguo/wiki/entities'
for fname in sorted(os.listdir(entities_dir)):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, encoding='utf-8') as f:
content = f.read()
original = content
lines = content.split('\n')
new_lines = []
seen_footnotes = {} # fn -> first line index
for line in lines:
fn_matches = re.findall(r'\^\[([^\]]+)\]', line)
if fn_matches:
for fn in fn_matches:
if fn in seen_footnotes:
# Duplicate — remove just the footnote marker from this line
line = re.sub(r'\^\[' + re.escape(fn) + r'\]', '', line)
line = line.rstrip()
if not line.strip():
line = None
break
else:
seen_footnotes[fn] = True
if line is not None:
new_lines.append(line)
if ''.join(new_lines) != content.replace('\n\n\n', '\n\n'):
content = '\n'.join(new_lines)
content = re.sub(r'\n{3,}', '\n\n', content)
with open(fpath, 'w', encoding='utf-8') as f:
f.write(content)
print(f"Deduplicated footnotes in {fixed} files")
Result: 46,172 duplicate footnote instances removed across 1,401 files (commit 77e88228).
Duplicate Wikilink Deduplication
Problem: 161 entity files have the same entity/concept wikilink appearing 2+ times (same [[entity-name]] in body text multiple times). Unlike footnotes, some repetition may be legitimate (same concept discussed in different sections), so only remove 3+ occurrences.
Fix:
import os, re
from collections import Counter
entities_dir = '/Users/jinguo/wiki/entities'
fixed = 0
for fname in sorted(os.listdir(entities_dir)):
if not fname.endswith('.md'): continue
fpath = os.path.join(entities_dir, fname)
with open(fpath, encoding='utf-8') as f:
content = f.read()
wikilinks = re.findall(r'\[\[([^\]|]+?)(?:\|[^\]]+)?\]\]', content)
entity_links = [w.strip() for w in wikilinks if not w.strip().startswith('raw/')]
link_counts = Counter(entity_links)
dups = {link: c for link, c in link_counts.items() if c > 2} # only 3+
for dup_link, count in dups.items():
pattern = r'\[\[' + re.escape(dup_link) + r'(?:\|[^\]]+)?\]\]'
matches = list(re.finditer(pattern, content))
if len(matches) > 2:
for m in reversed(matches[2:]): # keep first 2, remove rest
content = content[:m.start()] + '___RM___' + content[m.end():]
if '___RM___' in content:
content = content.replace('___RM___', '')
content = re.sub(r'\n{3,}', '\n\n', content)
with open(fpath, 'w', encoding='utf-8') as f:
f.write(content)
fixed += 1
print(f"Fixed wikilink duplicates in {fixed} files")
Result: 324 duplicate wikilink instances removed across 161 files (commit 77e88228).
NO FRONTMATTER — Missing Closing ---
Symptom: wiki-lint.mjs reports NO FRONTMATTER: entities/FILENAME even when a --- block appears to exist.
Root Cause: wiki-lint.mjs uses FM_RE = /^---\n([\s\S]*?)\n---/ — requires BOTH opening --- on its own line AND closing --- on its own line, with exactly one newline before and after each. If the closing --- is missing, or frontmatter ends directly with a YAML value without a trailing newline before the body, the regex fails to match.
Detection:
import os, re
for fn in os.listdir('entities'):
if not fn.endswith('.md'): continue
with open(f'entities/{fn}', encoding='utf-8') as f:
content = f.read()
fm_match = re.match(r'^---\n([\s\S]*?)\n---', content)
if not fm_match:
print(f"MISSING CLOSER: entities/{fn}")
elif not fm_match.group(1).strip():
print(f"EMPTY CLOSER: entities/{fn}")
Fix: Ensure proper frontmatter structure — the YAML block must have a closing --- on its own line, with at least one blank line between it and the first body heading:
---
title: ...
type: entity
sources:
- raw/articles/...
---
# Body Heading
Specific fixes:
- Missing closing
---: Add --- on its own line after the last YAML line
- No blank line after
---: Insert a blank line between closing --- and the first # Heading
- Trailing whitespace on
--- line: Ensure --- is alone with no trailing spaces
Real example fixed: claude-code-large-codebase-harness-configuration.md — closing --- was absent because the sources: list ended directly above the body heading. Fixed by inserting ---\n\n between sources block and heading (commit 9a25e616).
Self-Citation Entities — Body Copies from sources:
Symptom: Entity file where sources: [raw/articles/SLUG] and body is the raw article verbatim. Every paragraph gets ^[raw/articles/SLUG.md] self-citation. Lint flags 100% uncited paragraphs (EXCESS INFERRED), and Obsidian Local Graph shows hundreds of self-links.
Root cause: LLM synthesis task was essentially copy-paste with per-paragraph citation. The file IS the source with citations pointing to itself.
Fix: Delete the self-citation markers ^[raw/articles/SLUG.md] from the body. The frontmatter sources: already provides provenance. Per-paragraph self-citations add no value when the entire body comes from that source.
# Remove all self-citation markers from an entity
import re
with open(fp) as f: content = f.read()
# Remove self-citations: ^[raw/articles/SELF_SLUG.md]
content = re.sub(r'\s+\^\[raw/articles/SELF_SLUG\.md\]', '', content)
with open(fp, 'w') as f: f.write(content)
When NOT to fix: If the entity has genuine LLM synthesis (analysis, synthesis, original insights) layered on top of source citations, keep the citations — they serve a real provenance purpose.
Real case: entities/十年老技术开发的-ai-agent-探索之路.md — 352 self-citations removed, 0 remaining. Commit 49d2a4bf.
Duplicate Entity Variants — Same Article, Multiple Slugs
Symptom: One article stored as 5 entity files with different slugs (e.g., 十年老技术开发的-ai-agent-探索之路, 十年老技术开发的-ai-agent-探索之路-v2, ai-agent-exploration-legacy-dev, ai-agent-exploration-legacy-developer, ai-agent-exploration-path-legacy-tech). Each "Related entities" section links to the other 4 variants.
Impact: Confusing cross-linking, Obsidian Local Graph shows same-article self-references across variants, index.md has 5 entries for one article.
Detection:
# Group entity files by frontmatter title or first-line heading
import os, re
groups = {}
for fn in os.listdir('entities'):
if not fn.endswith('.md'): continue
with open(f'entities/{fn}') as f:
content = f.read()
m = re.search(r'^#\s+(.+)$', content, re.MULTILINE)
if m:
title = m.group(1).strip()
groups.setdefault(title, []).append(fn)
dups = {t: fs for t, fs in groups.items() if len(fs) > 1}
for t, fs in sorted(dups.items(), key=lambda x: -len(x[1])):
print(f"{t}: {len(fs)} variants → {fs}")
Strategy: Consolidate to one canonical entity. Other variants should either be deleted (if content is redundant) or merged into the canonical with proper differentiation.
Category 1: Bare Concept/Term Links (most common)
Entity files often contain [[Agent]], [[LLM]], [[OpenClaw]], [[Claude Code]], [[Codex]], [[Y Combinator]] etc. — these lint as BROKEN because no such wiki entry exists. Convert to plain text.
Category 5: Bare concepts/ / queries/ / comparisons/ Names (RESOLVED)
Symptom: 13 broken links across 6 files — wikilinks like [[agent-memory-lifecycle-philosophies]] resolve to entities/name.md which does not exist; the actual file lives in concepts/name.md.
Fix (string replace, not regex — hyphens in names break regex char classes on macOS):
broken_map = {
'agent-memory-lifecycle-philosophies': 'concepts/agent-memory-lifecycle-philosophies',
'agent-memory-system-design': 'concepts/agent-memory-system-design',
'context-management-agent-systems': 'concepts/context-management-agent-systems',
'autonomous-agent-systems': 'concepts/autonomous-agent-systems',
'multi-agent-systems': 'concepts/multi-agent-systems',
'inference-optimization': 'concepts/inference-optimization',
'agent-evaluation-benchmark-frameworks': 'concepts/agent-evaluation-benchmark-frameworks',
'openclaw-architecture': 'concepts/openclaw-architecture',
}
for fname, replacements in source_files.items():
content = open(f'entities/{fname}').read()
for broken_name, prefix in broken_map.items():
content = content.replace(f'[[{broken_name}]]', f'[[{prefix}]]')
content = content.replace(f'[[{broken_name}|', f'[[{prefix}|')
open(f'entities/{fname}', 'w').write(content)
Resolved: commit ef694e6a — all 13 links fixed.
Category 6: Wikilink with .md Extension
Symptom: 2 files had wikilinks with erroneous .md suffix (e.g. [[queries/llm-training-rl-research.md]]). Obsidian appends .md again → broken.
Fix: String remove .md before the closing ]] or |.
# Remove .md suffix from wikilinks
content = content.replace('[[queries/llm-training-rl-research.md]]',
'[[queries/llm-training-rl-research]]')
Resolved: commit ef694e6a.
Category 7: Plain-Text Bullets in ## 相关实体 Sections
Symptom: Entity files have ## 相关实体 sections where bullets are plain text (e.g. - Anthropic 联创：2028 年实现 AI 自我构建的概率超过 60%) rather than [[entities/slug|显示名]] wikilinks. These produce zero bidirectional graph connections — Obsidian shows no backlinks, no graph edges.
Detection:
import re, os
base = "/path/to/wiki/entities"
broken_per_file = {}
for fname in os.listdir(base):
if not fname.endswith(".md"): continue
fpath = os.path.join(base, fname)
with open(fpath, encoding="utf-8") as f:
lines = f.readlines()
in_rel = False
for i, line in enumerate(lines):
if "## 相关实体" in line or "## Related" in line or "## 相关页面" in line:
in_rel = True; continue
if not in_rel: continue
if line.startswith("## ") or line.startswith("# "): in_rel = False; continue
stripped = line.strip()
if not stripped or stripped.startswith(">"): continue
if stripped.startswith("-") and "[[" not in stripped:
broken_per_file.setdefault(fname, []).append((i+1, stripped[:100]))
Common patterns:
| Pattern | Example | Fix |
|---|
| Recurring article title (318 occurrences, 12 texts across 42 files) | - Anthropic 联创：2028 年实现 AI 自我构建的概率超过 60% | Map to existing entity slug, replace with [[entities/slug|title]] |
| Empty bullet | - or - | Remove |
| Em-dash sub-description | - — tool注册表元数据与实际行为间的验证断层 | Remove |
| Section divider | --- or -- | Remove |
| Bare URL | - 控制台：https://... | Remove |
| Company/org bullet with description | - **Anthropic** — 治理者岗位占比最高的公司之一 | Remove if no entity exists |
| Code field item | - `requirements`：精确关联到 REQUIREMENTS.md | Remove |
| RSS navigation item | - - 次日AI速递 | Remove |
| Deleted topic map reference | - ai industry news topic map（已删除） | Remove |
Fix strategy (in this order):
- Build text→slug mapping by searching entity filenames and frontmatter titles for each plain text
- Bulk replace the 12 high-frequency recurring texts (318 occurrences across 42 files):
item_map = [
("Anthropic 联创：2028 年实现 AI 自我构建的概率超过 60%",
"anthropic-联创2028-年实现-ai-自我构建的概率超过-60"),
# ... all 12
]
for text, slug in item_map:
pattern = re.compile(r'^(- ' + re.escape(text) + r')$', re.M)
replacement = f'- [[entities/{slug}|{text}]]'
new_content = pattern.sub(replacement, content)
- Bulk remove empty bullets, em-dash continuations, dividers, URLs, RSS nav items, deleted refs — one pass per file
- Remaining orphans — find entity slug via title search, or remove if no entity exists
Real outcome: 318 high-freq text fixes + 122 removals across ~180 entity files → 0 lint errors.
Key lesson: Plain-text related-entity bullets are invisible in the graph. Well-linked body text does NOT protect against unlinked ## 相关实体 bullets — always scan that section specifically.
Common bare links that should be plain text:
- Company/org names without a wiki page:
[[Y Combinator]] → Y Combinator
- Generic terms:
[[Agent]] → Agent, [[LLM]] → LLM, [[RAG]] → RAG
- Product names without wiki pages:
[[OpenClaw]] → OpenClaw, [[Claude Code]] → Claude Code
- Tool names:
[[jq]] → jq, [[FFmpeg]] → FFmpeg, [[SQLite]] → SQLite
- Academic institutions:
[[Stanford University]] → Stanford University
Python fix pattern:
from pathlib import Path
f = Path("/path/to/entity.md")
text = f.read_text()
bare_terms = ["Agent", "LLM", "OpenClaw", "Claude Code", ...]
for term in bare_terms:
old = f"[[{term}]]"
if old in text:
text = text.replace(old, term)
print(f"OK: {term}")
f.write_text(text)
Category 2: Entity References to Deleted/Non-existent Files
When an entity file references another entity that was deleted or never existed (e.g., [[entities/claude-code-开发负责人...]]), the lint reports a BROKEN LINK. Convert to plain text describing the reference, or redirect to a similar existing entity.
# Replace broken entity wikilink with plain text
text = text.replace(
"[[entities/deleted-entity]]",
"deleted-entity description"
)
Category 3: Footnote Refs with Wikilink Brackets
[[^4]], [[^7]] etc. appearing in text — footnote references accidentally wrapped in [[]]. Remove brackets:
text = text.replace("[[^4]]", "[^4]")
Category 4: WL_RE False Positive — Code Literals with [[...]]
Symptom: BROKEN LINK: entities/X -> [["macd", "rsi_14", ...]] — Python/JavaScript list/dict literals inside code blocks matched as wikilinks.
Root cause: WL_RE = /\[\[([^|\]#]+?)(?:\|[^\]]*?)?(?:#[^\]]*?)?\]\]/g matches [[...]] anywhere, including code.
Fix: Restructure code to avoid the [[...]] pattern:
# Bad
indicators = stock_df[["macd", "rsi_14", ...]]
# Good — use .loc or list() wrapper
indicators = list(stock_df.columns[["macd", "rsi_14", ...]])
# Or
indicators = stock_df.loc[:, ["macd", "rsi_14", ...]]
Finding All Broken Links — Full Audit Pattern
node scripts/wiki-lint.mjs . 2>&1 | grep "BROKEN LINK" | sed 's/.* -> //' | sort | uniq -c | sort -rn
This shows the broken target and count. For each unique broken target:
- Check if the target exists as a file:
ls entities/TARGET.md or ls concepts/TARGET.md
- If exists → fix the path in the entity file
- If NOT exists → replace
[[TARGET]] with plain text
Pitfall: MOC Link Corruption from Regex Fix Scripts
Symptom: After running a Python script to fix broken links in MOCs, links like [[entities/slug]] become [[entitiesslug]] (missing / separator) or [[conceptsslug]].
Root cause: Regex replacement using re.sub(r'\[\[(entities|concepts)/([^|\]]+?)(?:\|([^\]]+))?\]\]', fix_link, content) — the fix_link function returns a reconstructed string but accidentally drops the / between prefix and slug.
Prevention: When writing fix_link functions for MOC wikilinks, ALWAYS include the / in the return value:
def fix_link(m):
prefix = m.group(1) # "entities" or "concepts" (WITHOUT trailing /)
slug = m.group(2)
display = m.group(3)
# ...
return f"[[{prefix}/{new_slug}|{display}]]" # ← MUST include /
Detection: After any bulk MOC link fix, run:
grep -r '\[\[entities[a-z]' moc/ | head -5 # Missing / after "entities"
grep -r '\[\[concepts[a-z]' moc/ | head -5 # Missing / after "concepts"
Recovery: Add the missing / back:
content = re.sub(r'\[\[(entities|concepts)([a-z])', r'[[\1/\2', content)
Batch Citation Fix — Two Sources Formats
The existing fix-excess-inferred-v5.mjs script handles the standard case. But when doing ad-hoc Python citation fixes, entities use TWO different sources: formats:
Format 1 — YAML list (multi-line):
sources:
- raw/articles/slug-a
- raw/articles/slug-b
Format 2 — Inline array (single-line):
sources: [raw/articles/slug-a, raw/articles/slug-b]
A citation script that only handles Format 1 will miss ~80% of entities. Always parse both:
sources = []
# Format 1: YAML list
list_match = re.search(r'sources:\s*\n((?:\s+-\s+.+\n?)+)', fm)
if list_match:
sources = [s.strip() for s in re.findall(r'-\s+(.+)', list_match.group(1))]
# Format 2: inline array
if not sources:
inline_match = re.search(r'sources:\s*\[([^\]]+)\]', fm)
if inline_match:
sources = [s.strip() for s in inline_match.group(1).split(',')]
Wiki Evaluation Methodology
