wiki-quality-improvement

name	wiki-quality-improvement
description	Systematically improve wiki entity quality through phased optimization: clean stubs, backfill scores, expand thin content, consolidate tags, build synthesis pages.
references	["references/wiki-quality-diagnostic.md","references/wiki-entity-wikilink-patterns.md"]

Wiki Quality Improvement

Turn a raw wiki archive into a "second brain" think-tank through phased optimization.

Assessment Baseline (as of 2026-05-18, after Phase 1+2)

Post-cleanup: 2313 pages, 1104 entities, 100% have review_value (69% scored 7-9).

Updated baseline (2026-06-29): 2498 entities, 30.9/30 avg quality score, 6 entities below 20 (needs work), 0 below 15 (priority). Quality review script at scripts/wiki-quality-review.py.

Quality Review Script

A reusable quality review script is available at scripts/wiki-quality-review.py. It scores all entities on a 30-point scale across 4 dimensions:

Dimension	Points	Checks
Frontmatter	10	title, description, created, updated, type, tags, confidence, review_value
Sections	10	深度分析 (≥3 subsections), 实践启示, 相关实体, provenance backlink
Depth	10	body size, entity links (≥3), paragraphs (≥3)
Citations	5	^[raw/articles/] or [[raw/articles/]] references

Usage: python3 scripts/wiki-quality-review.py --top=15

Cron integration: Schedule weekly with bash ~/wiki/scripts/cron-wiki-quality-review.sh (silent when all entities score ≥20).

Updated baseline (2026-05-25): 1552 entities, body length distribution right-skewed (949 in 2k-5k, 77 >10k). 701 entities have 0 entity wikilinks (only raw/ links). 19 entities <1.2k chars are delete/merge candidates. 3 lint errors remain (false positives from code block examples in concepts/wiki-audit-skill.md).

Quality Thresholds

Tier	Criteria
OK	review_value >= 7, body > 2KB, has entity wikilinks
Needs Entity Links	body > 1KB but 0 entity wikilinks
Needs Expansion	review_value >= 4, body < 2KB
Needs Score	body > 800B but missing review_value
Stub	review_value 1-3, body < 800B (delete/merge candidate)

Phases

Phase 1: Concept Layer Expansion

Target: Convert all concept skeletons to rich synthesis pages.

Priority queue (execute_code to generate):

from pathlib import Path
import re

def count_entity_links(path):
    text = path.read_text(errors='replace')
    return len(re.findall(r'\[\[entities/', text))

wiki = Path("/Users/jinguo/wiki/concepts")
results = []
for f in sorted(wiki.glob("*.md"), key=lambda x: x.stat().st_size):
    size = f.stat().st_size
    links = count_entity_links(f)
    results.append((size, links, f.name))

# Priority: thin+low-links first
for size, links, name in results:
    label = "SKEL" if size < 2000 else ("LOW" if links < 3 else "RICH")
    if label != "RICH":
        print(f"{label} {size:>6}B  {links:>3}links  {name}")

Decision rules:

• <2KB skeleton → full expansion: add sections, entity links, provenance
• 2-5KB with 0-2 entity links → lightweight: add 3-5 [[entities/X]] links + 1 new section if content is solid
• >5KB with 3+ entity links → already rich, skip

Subagent pattern for lightweight expansion (3 files/batch):

1. Read concept file
2. Search entities/ for matching tags/titles: search_files(pattern, path="entities", target="files")
3. Add 3-5 [[entities/slug]] wikilinks at natural integration points
4. If <5KB, add 1 substantive section (not padding)
5. Update frontmatter: updated: YYYY-MM-DD, add related: if missing
6. Run lint, commit each file separately

Phase 1 Concept Expansion: Realistic 2026-06-18 Patterns

IMPORTANT — Phase 1 thresholds above (2026-05-25) are STALE for the current wiki: After the 2026-06-18 wiki audit (mass entity wikilink backfill + 7 MOC creation), the candidate landscape shifted dramatically. The <2KB + 0-2 links filter now catches only 1 concept (llm-artifact-optimization). Most thin concepts already have 6-12 entity links from earlier bulk enrichment, but still need content depth, not just interlinking.

Realistic candidate classification (2026-06-18 wiki state, 172 concepts):

Class	Size	Pattern	Count	Action
A — Skeleton	≤3KB	Often has TODO placeholders, missing 核心定义/实践启示	~60	Subagent full expansion OR manual pre-skeleton + subagent fill
B — Weak link	3-8KB, ent_links < 8	Structure complete, few cross-references	~17	Subagent add 5-8 entity links
C — Near-target	8-10KB	Almost at 10KB golden line	~17	Subagent add 5-10 entity links (skip ## 深度分析 if < 8KB)

Trial results (2026-06-18, 5 concepts, commit b7284b04):

File	Before	After	Mode	Outcome
model-distillation-compression	1.4KB	7.8KB	A (subagent)	Subagent succeeded despite max_iter risk
llm-artifact-optimization	3.1KB	6.6KB	B (manual)	Subagent failed; manual ## 深度分析 add (3 sub-sections, 3.5KB) faster than retrying subagent
source-first-knowledge-compilation	6.8KB	11.0KB	C (subagent)	Subagent succeeded, +8 entity links + ## 深度分析
harness-context-window-management	8.8KB	11.1KB	C (subagent)	Subagent succeeded, +7 entity links
agent-engineering-capability-map	8.6KB	10.2KB	C (subagent)	Subagent succeeded, +11 entity links

Success rates by class (5-trial sample):

• C class (6-9KB add-links only): 3/3 = 100% success. Subagent context is small, plenty of room for verification.
• B class (3-8KB add-links + maybe 1 section): not tested in this trial, expected 80%+ based on C extrapolation.
• A class (≤3KB from-scratch): 1/2 = 50% success rate. Subagent task 1 hit max_iterations HTTP 500 on the second file — but had already written the first file before failure. ~50% probability of writing at least one of the batched files before failing.

Pitfall — A class requires pre-skeleton for reliability: Subagents writing from-scratch content for ≤3KB concept pages run out of iterations before completion. Pre-generate a skeleton with TODO sections using patch, then dispatch subagent to fill content. Sample skeleton:

## 深度分析

### 1. 第一原理
<!-- TODO -->

### 2. 关键设计权衡
<!-- TODO -->

### 3. 实战应用
<!-- TODO -->

## 实践启示

### 1. 工程师视角
<!-- TODO -->

### 2. 团队配置视角
<!-- TODO -->

Then subagent prompt: "Skeleton already in place. Fill each TODO with 2-3 paragraphs of substantive Chinese content. DO NOT add raw citations. Verify entity links exist."

Pitfall — Subagent failure recovery without losing work: When delegate_task returns status="failed", exit_reason="max_iterations":

1. Do NOT assume all files are unchanged — the subagent may have written some files before failing.
2. Check git status for M concepts/ entries — those are your recoverable artifacts.
3. Read each modified file to verify content quality (not just size delta).
4. For the missing file, either retry the subagent or do manual expansion (whichever is faster — manual is faster for ≤4KB additions).

Manual patch is faster than subagent for ≤4KB additions:

• Subagent: ~2-3 minutes per file (read + search + write + verify)
• Manual patch: 30-60 seconds per file (read + draft + patch + lint)
• For adding just ## 深度分析 with 3-4 subsections (~3KB), manual is 3-4x faster and equally reliable.

Updated subagent prompt template (2026-06-18, A class with pre-skeleton):

Wiki root: /Users/jinguo/wiki. Format: YAML frontmatter + markdown body. Write in Chinese.

File: {concept_path} (current size: {size_kb}KB)
Skeleton has been pre-generated with TODO placeholders. Fill each TODO with 2-3 substantive paragraphs.

Strict rules:
- DO NOT add ^[raw/articles/...] citations. These concepts have raw_refs=0; no verified sources.
- Only add [[entities/...]] and [[concepts/...]] wikilinks. NEVER [[topics/...]].
- Verify ALL wikilink targets exist before adding (use: ls /Users/jinguo/wiki/entities/SLUG.md).
- Use write_file to overwrite the entire file (patch tool can corrupt Chinese wikilinks).
- Bump frontmatter `updated:` to {today}.
- Preserve ALL existing frontmatter and content. Only FILL TODO sections.

After writing, run:
  cd /Users/jinguo/wiki && node scripts/wiki-lint.mjs /Users/jinguo/wiki 2>&1 | grep -cE "^ERROR"
Report the result. Expect 0.

Report back: final file size, lint error count, any wikilinks you couldn't verify.

Updated subagent prompt template (2026-06-18, B/C class add-links):

Wiki root: /Users/jinguo/wiki. Format: YAML frontmatter + markdown body. Write in Chinese.

File: {concept_path} (current size: {size_kb}KB, ent_links: {n})

Strategy: add entity interlinks to make the concept graph denser. File structure is already good.

Strict rules:
- DO NOT add ^[raw/articles/...] citations.
- Add 5-10 [[entities/SLUG|显示名]] wikilinks at natural integration points.
- For C class (8-10KB): skip ## 深度分析 add — just interlinking is enough.
- For B class (3-8KB): if <8KB after linking, add a small ## 深度分析 section (2-3 subsections).
- Verify ALL wikilink targets exist (ls /Users/jinguo/wiki/entities/SLUG.md).
- Use write_file to overwrite the entire file.
- Bump frontmatter `updated:` to {today}.

After writing, run:
  cd /Users/jinguo/wiki && node scripts/wiki-lint.mjs /Users/jinguo/wiki 2>&1 | grep -cE "^ERROR"
Report the result. Expect 0.

Report back: final file size, lint error count, list of new entity links added.

Batch sizing (3-way split, max_concurrent_children=3):

• A class: 1-2 concepts per subagent (subagent context can balloon with from-scratch writing)
• B/C class: 2-3 concepts per subagent (lighter work)
• 3 subagents parallel × 2-3 concepts each = 6-9 concepts per batch
• 5-7 batches covers all remaining B/C class (~34 concepts) — ~2-3 hours

execute_code is BLOCKED in cron_mode — use terminal + Python script (write_file to /tmp first, then python3 /tmp/script.py). This applies to ALL wiki-quality-improvement workflows when running inside cron jobs.

Duplicate concept files to clean up (2026-05-21):

• harness-component-expiry-build-to-delete.md ← expanded (kept)
• harness-component-expiry-and-build-to-delete.md ← 5.3KB, 0 links (delete, redirect links to the kept version)

Phase 1: Entity Wikilink Density — Fix 701 Zero-Link Entities

2026-05-25 finding: 701/1552 entities (45%) have 0 entity wikilinks — they only link to raw articles, not to other entities. This is the largest quality gap in the current wiki.

2026-06-10 proven solution — automated tag+keyword Counter matching: When zero-link count is in the hundreds, manual per-entity linking doesn't scale. The proven automated approach:

1. Build slug_tags dict: {slug: [tag1, tag2, ...]} from frontmatter
2. Build slug_title dict: {slug: title_lowercase}
3. For each zero-link entity, compute Counter of tag-overlap scores against all other entities
4. Also compute title-keyword overlap (re.findall(r'[a-z]{3,}', title))
5. Take candidates.most_common(5), verify targets exist on disk, add [[entities/slug]] wikilinks
6. Result: 921 entities fixed in a single batch, 0 broken links

See references/automated-zero-link-fix-2026-06-10.md for the full script pattern.

Priority batch (top 20 lowest quality by score):

1. letsdatascience-igor-babuschkin-...-8c36ce09 (721 chars, score=3.5) — duplicate, mark for deletion
2. 今天起-claude正式接入office全家桶-... (749 chars, score=3.5, tags=[default]) — delete candidate
3. fastlane-create-winning-short-form-content-in-seconds (833 chars, score=3.7, tags=[default]) — delete candidate
4. every-ai-subscription-is-a-ticking-time-bomb-for-enterprise (945 chars) — add entity links to OpenAI/Anthropic/vendor-lock-in entities
5. build-live-translation-apps-with-gpt-realtime-translate (946 chars) — add entity links to OpenAI/gpt-realtime/translation entities
6. generalization-dynamics-of-lm-pre-training-jiaxin-wen-1 (1124 chars) — add entity links to llm/evaluation/research entities

Batch pattern (20-30 entities per session):

1. Generate quality score for all 701 zero-entity-wl entities using references/wiki-quality-diagnostic.md formula
2. Sort by score ascending, take top 20-30
3. For each entity: identify topic keywords → find matching entity files via search_files(pattern, path="entities", target="files") → add 3-5 [[entities/slug]] wikilinks at natural integration points
4. Update updated: date in frontmatter
5. Run lint, commit each file separately

Bulk Tag+Keyword Cross-Referencing (2026-06-10 — 921 entities fixed)

When zero-entity-wikilink count is very high (900+), per-entity search_files is too slow. Use a bulk algorithm instead:

Algorithm: Build tag→slug index, then for each zero-link entity, find candidates via tag overlap + title keyword overlap.

import os, re
from collections import Counter, defaultdict

# 1. Build tag→slug index
tag_index = defaultdict(set)
slug_tags = {}
slug_title = {}

for fn in os.listdir(entities_dir):
    if not fn.endswith('.md'): continue
    fp = os.path.join(entities_dir, fn)
    with open(fp, 'r', encoding='utf-8', errors='replace') as f:
        c = f.read()
    slug = fn[:-3]
    tag_match = re.search(r'^tags:\s*\[([^\]]*)\]', c, re.MULTILINE)
    if tag_match:
        tags = [t.strip().lower() for t in tag_match.group(1).split(',')
                if t.strip() and t.strip() != 'default']
        slug_tags[slug] = tags
        for t in tags:
            tag_index[t].add(slug)
    title_match = re.search(r'^title:\s*["\']?([^"\'\n]+)["\']?', c, re.MULTILINE)
    if title_match:
        slug_title[slug] = title_match.group(1).lower()

# 2. For each zero-link entity, find candidates via tag overlap + keyword overlap
for fn in os.listdir(entities_dir):
    if not fn.endswith('.md'): continue
    fp = os.path.join(entities_dir, fn)
    with open(fp, 'r', encoding='utf-8', errors='replace') as f:
        c = f.read()
    slug = fn[:-3]
    elinks = [l for l in re.findall(r'\[\[entities/([^]|]+)', c) if l.strip() != slug]
    if len(elinks) > 0: continue  # Already has links

    my_tags = slug_tags.get(slug, [])
    my_title = slug_title.get(slug, slug.replace('-', ' '))

    candidates = Counter()
    for t in my_tags:
        for other in tag_index[t]:
            if other != slug:
                candidates[other] += 1

    title_words = set(re.findall(r'[a-z]{3,}', my_title))
    for other_slug, other_title in slug_title.items():
        if other_slug == slug: continue
        other_words = set(re.findall(r'[a-z]{3,}', other_title))
        overlap = title_words & other_words
        if len(overlap) >= 2:
            candidates[other_slug] += len(overlap)

    top = candidates.most_common(5)
    # Add "## 相关实体" section with verified wikilinks
    links_to_add = []
    for cslug, score in top:
        if os.path.exists(os.path.join(entities_dir, f"{cslug}.md")):
            links_to_add.append(f"- [[entities/{cslug}]]")
    if links_to_add:
        # Insert before raw link or end of file
        section = "\n## 相关实体\n" + "\n".join(links_to_add) + "\n"
        # ... insert into content ...

Results: 921 zero-link entities fixed in 2 batches (50 + 871), zero-entity-wikilink rate dropped from 49% to 1%. Zero broken links verified by lint.

Key insight: Tag overlap is the strongest signal; title keyword overlap catches additional matches but has more false positives. Combined scoring (tag weight + keyword weight) works well.

Piped wikilink pitfall: When fixing broken wikilinks, bare [[entities/slug]] can be fixed with .replace(), but piped [[entities/slug|Display Text]] requires regex: re.sub(r'\[\[entities/slug\|[^\]]+\]\]', '**Display Text**', content).

Subagent prompt template:

Read entity: {entity_path}
Find matching entity files in entities/ that share tags or topic keywords: {keywords}
Add 3-5 [[entities/slug]] wikilinks at natural positions in the body (before relevant sections or in a new "## 相关实体" section)
Verify all wikilinks resolve before writing.
Update frontmatter: updated: YYYY-MM-DD

2026-05-25 duplicate entity pattern:

• letsdatascience-igor-babuschkin-seeks-up-to-1-billion-for-river-ai-8c36ce09 (721 chars, tags=[ai, funding, river-ai]) — short summary of the richer igor-babuschkin-seeks-up-to-1-billion-for-river-ai (5.9KB, 0 entity wikilinks). Same title, same content, 8× shorter. Delete the short one, link references to the richer version.
• fastlane-create-winning-short-form-content-in-seconds (833 chars, tags=[default]) — thin content, no AI technical depth, just a product page summary. Delete candidate.
• 今天起-claude正式接入office全家桶-跨应用还能共享记忆-1dbbc8 (749 chars, tags=[default]) — pure news, no AI technical insight. Delete candidate.

Rule for duplicate detection: Same title prefix + one version is ≥3× longer = likely duplicate. Always keep the longer, richer version.

Phase 2: Backfill review_value Scores

Use batch-entity-scoring skill for bulk scoring (500+ entities):

1. Auto-score entities < 800B as 1/10 (clearly stubs)
2. Score remaining by body quality: headings, wikilinks, paragraphs, body length
3. Insert review_value before closing --- of frontmatter (NOT after tags: — fails on non-standard frontmatter)
4. Subagent LLM scoring only for entities >= 800B with real content

Phase 3: Expand Thin Entities

For entities with review_value >= 4 but body < 2KB. Priority: highest score, thinnest body first.

2026-05-23 finding — all <50w entities are NOT stubs: Re-audit showed all 35 entities below 50 words contained ## 深度分析 sections with real content. Word count was low due to CJK character encoding (each Chinese char = 1 word but counted by UTF-8 byte division). 0 entities required deletion — only 1 candidate for manual expansion (claude-code-demand-research-taosecho, 38w, 1 paragraph).

Rule: Before bulk-deleting thin entities, read a sample body to check for ## 深度分析 sections. Count-based stubs may be encoding artifacts. Use body content scan, not byte-length heuristics.

Subagent expansion pattern (2-3 entities/batch):

1. Read entity + raw article
2. Add ## 深度分析 — 3-6 subsections synthesizing beyond source
3. Add ## 实践启示 — actionable advice segmented by audience
4. Add paragraph citations: ^[raw/articles/slug.md:LINE_RANGE]
5. CRITICAL: verify ALL cross-reference wikilinks exist before adding (subagents are over-eager)
6. Update updated date, preserve existing content, use write_file

Expansion spec for 深度分析 + 实践启示 (2026-06-04 refinement):

• Position: Both sections go BEFORE the → [[raw/articles/...|原文存档]] line
• Structure: ## 深度分析 (5 subsections) + ## 实践启示 (5 subsections)
• Target body size: 4500–7000 chars (CJK chars each count as 1; measure with python3 -c "with open(f) as fh: content=fh.read(); body=content[content.find('\n\n# '):]; print(len(body))")
• Line citations: Every substantive claim needs ^[raw/articles/SLUG.md:LINE_RANGE] — use grep -o to count actual citations, not python regex (regex trips on ^- ranges)
• Wikilinks: Only [[entities/...]] and [[concepts/...]] — NEVER [[topics/...]] (topics/ is not a wiki section). Verify all targets exist with ls entities/ | grep -i SLUG before adding
• File tool: Use write_file (NOT patch) — patch corrupts Chinese filenames on macOS. write_file overwrites cleanly
• Frontmatter bump: Always update updated: to today's date when expanding
• Section naming: Use Chinese headings (## 深度分析, ### 3.1 中文副标题) — domain-specific, not generic templates
• Practice启示 specificity: Must be actionable and domain-specific to the entity's domain (e.g., manufacturing QC, not generic "best practices")

Phase 3B: Quality Repair

After expansion, audit all entities:

1. Fix broken wikilinks (verify each [[entities/X]] exists)
2. Add citations to entities with 0 provenance markers
3. Add line-number citations where missing

Phase 4: Tag Consolidation

Compress tag space from hundreds of unique tags to a focused ontology.

Tag audit pattern (detect real problems vs parser artifacts):

import os, re, yaml
from collections import Counter

wiki_path = "/Users/jinguo/wiki/entities"

# CORRECT YAML list parsing — use PyYAML, not regex
tag_counter = Counter()
files_with_inline_tags = 0
files_with_list_tags = 0
empty_tag_files = []

for root, dirs, files in os.walk(wiki_path):
    for f in files:
        if not f.endswith('.md'):
            continue
        fp = os.path.join(root, f)
        with open(fp, errors='replace') as fh:
            content = fh.read()
        fm_match = re.search(r'^---\n(.*?)\n---\n', content, re.DOTALL)
        if not fm_match:
            continue
        try:
            fm = yaml.safe_load(fm_match.group(1) + '\n')
            tags = fm.get('tags', [])
            if isinstance(tags, list):
                files_with_list_tags += 1
                for t in tags:
                    if t:  # skip None/empty
                        tag_counter[t] += 1
                if len(tags) == 0:
                    empty_tag_files.append(f)
            elif isinstance(tags, str):
                files_with_inline_tags += 1
                tag_counter[tags] += 1
        except:
            pass

# Check for 'default' tag — these are placeholders with no semantic value
# Fix: delete 'default' tag entries manually

Common tag quality issues:

Issue	Count	Fix
tags: [] (empty array)	~200 files	Auto-tag: title keywords + source field + body keywords
Duplicate tags: [] keys	~26 files	Remove duplicates (YAML last-value-wins but makes frontmatter dirty)
'default' tag	~22 files	Manual removal — no semantic value
: (empty string from bad regex parsing)	0	Parser artifact, not real data

Auto-tag strategy for tags: [] files:

# Priority: source field > title keywords > body keywords
tag_rules = {
    'source contains wechat': ['wechat'],
    'source contains newsletter': ['newsletter'],
    'source contains aws': ['aws'],
    'source contains github': ['open-source'],
    'title has agent': ['agent'],
    'title has llm/语言模型': ['llm'],
    'title has mcp': ['mcp'],
    'title has memory': ['memory'],
    'title has 安全/security': ['security'],
    'title has 架构/architecture': ['architecture'],
    'title has multi-agent': ['multi-agent'],
    'title has claude-code': ['claude-code'],
    'title has anthropic': ['anthropic'],
    'title has openai': ['openai'],
    'title has deepseek': ['deepseek'],
    'title has 评测/benchmark': ['evaluation'],
    'title has inference': ['inference'],
    'title has 视频/video': ['video'],
    'title has harness': ['harness-engineering'],
    'title has skill': ['skill'],
    'title has rag': ['rag'],
}

Bulk auto-tagging implementation (2026-06-10 — 22 entities fixed):

When tags: [] or tags: [default] count is moderate (<50), use keyword-matching against title+slug:

tag_rules = {
    'agent': ['agent'], 'harness': ['harness-engineering'], 'skill': ['skill'],
    'claude': ['claude'], 'openai': ['openai'], 'anthropic': ['anthropic'],
    'deepseek': ['deepseek'], 'mcp': ['mcp'], 'llm': ['llm'],
    'aws': ['aws'], 'amazon': ['aws'], 'security': ['security'], '安全': ['security'],
    'memory': ['memory'], 'rag': ['rag'], 'multi-agent': ['multi-agent'],
    'eval': ['evaluation'], 'benchmark': ['evaluation'], 'architecture': ['architecture'],
    'inference': ['inference'], 'fine-tun': ['fine-tuning'], 'video': ['video'],
    'nvidia': ['nvidia'], 'netflix': ['netflix'], 'google': ['google'],
    'microsoft': ['microsoft'], 'coding': ['ai-coding'], 'cost': ['finops'],
    'prompt': ['prompt-engineering'], 'sandbox': ['sandbox'], 'deployment': ['deployment'],
}

for fn in os.listdir(entities_dir):
    # ... read content, check if tags: [] or tags: [default] ...
    title = (title_match.group(1) if title_match else slug).lower()
    search_text = title + ' ' + slug.replace('-', ' ')
    new_tags = set()
    for keyword, tag_list in tag_rules.items():
        if keyword in search_text:
            for t in tag_list: new_tags.add(t)
    if not new_tags: new_tags = {'uncategorized'}
    # Replace tags: [] → tags: [tag1, tag2] in content

Result: 6 empty-tag + 16 default-tag entities auto-tagged. Remaining 31 empty-tag entities need manual review (no keyword matches in title/slug).

YAML list format — always use:

tags: [tag1, tag2, tag3]

NOT inline list tags: [tag1, tag2] with other keys below (can cause parsing issues).

Phase 4: Tag Consolidation (continued)

Phase 5: Synthesis Pages

Build connection pages linking related entities with original insights.

Post-Deletion Cleanup

After deleting entities, clean three file types:

1. index.md — remove entries for deleted slugs
2. queries/*.md — remove lines referencing deleted entities
3. entities/*.md — remove broken wikilinks (match both [[entities/slug]] and [[entities/slug|Display]])

Pitfalls

• Subagents add broken wikilinks if not instructed to verify. Always include "verify ALL cross-references exist before adding" in subagent prompts.
• Many expanded entities lack line-number citations. Acceptable for LLM synthesis, but substantive source claims need line numbers.
• Practice advice sections tend toward generic templates. Make them domain-specific.
• Always regenerate entity slug lists from disk before spawning subagents (stale lists cause mismatches).
• Insert frontmatter fields before closing ---, not after tags:.
• Check for same-topic duplicate slugs before expansion.
• Broken raw citations accumulate silently: After expansion, always run node scripts/wiki-lint.mjs . 2>&1 | grep "BROKEN" and fix before committing. 598 broken ^[raw/articles/...] citations were found in one session — entities referencing raw files that don't exist on disk. Prevention: verify raw slug exists before adding citations.
• Piped wikilinks require regex fix: [[entities/slug|Display Text]] can't be fixed with .replace(). Use re.sub(r'\[\[entities/slug\|[^\]]+\]\]', '**Display Text**', content).
• Lint-verify-commit cycle is mandatory: After every expansion batch: lint → fix broken links/citations → verify lint=0 → commit. Skipping verification lets broken references compound across batches.

Batch quality review pattern — v×c<30 with full body read (2026-05-20/21 two-batch session)

Workflow: Scan all entities for v×c < 30 → read body preview (first 1200 chars) → classify:

• v=0, c=0: Never reviewed — read body, score or delete
• c=0: review_value set but confidence never scored — need confidence fill
• Low score with real content: keep; with non-AI/ML content: delete

Dedup + index cleanup workflow (mandatory before every commit)

After deleting entities, scan for these patterns BEFORE committing:

1. index.md duplicate raw article slugs: scan for same raw/articles/SLUG appearing multiple times, keep first, remove later duplicates
2. queries/*.md broken entity links: remove references to deleted entity slugs
3. sed fix after patch operations: After ANY patch tool call on entity files, run:

   cd <wiki_root> && sed -i '' 's/^|- /- /' entities/*.md
   

   The patch tool corrupts frontmatter list items (inserts |- instead of - ). This affects entities/*.md in addition to index.md/log.md.

Adding entity wikilinks to zero-wikilink entities (2026-05-25)

Target: 701 entities with 0 entity wikilinks (only raw/ links).

Workflow per entity:

1. Identify topic keywords from title + first paragraph
2. Verify potential target entities exist: ls entities/ | grep -i KEYWORD
3. Add ## 相关实体 section at end with 3-5 [[entities/slug]] wikilinks
4. Never add wikilinks to non-existent entities (causes broken links that lint catches)
5. For entity-typed terms (Llama, Mistral, vendor-lock-in) that don't exist as entities → keep as plain text, not wikilink syntax

Validated entity targets (2026-05-25, 19 entities updated):

Topics → Valid entity targets:
complexity/software-design → 10-common-component-architecture-mistakes, agent-harness-architecture-deep-dive-aksahy
sovereign-cloud/cloud → alibaba-cloud-cio-ai-productivity-reframe, ai-chip-architecture-first-principles
security/anthropic → ai-agents-security-survey-attack-defense, anthropic-12-mcp-production-patterns
claude/prompts/engineering → claude-code-prompt-context-harness, harness-generator-evaluator-anthropic
diffusion/LLM/he-kaiming → karpathy-ai-agent-7-bits-value-decline, ai-chip-architecture-first-principles
multimodal/vision/minicpm → ai-chip-architecture-first-principles, minicpm-v-46-13b-xinazhiyuan
aws/infrastructure/cdk → ai-agents-security-survey-attack-defense, agent-harness-architecture-deep-dive-aksahy
bedrock/agent/business-intelligence → aws-bedrock-multi-agent-collaboration-guide, agent-harness-architecture-deep-dive-aksahy
agent-auto-improvement → agent-harness-architecture-deep-dive-aksahy, aws-bedrock-multi-agent-collaboration-guide
agent/rag/protocol-h → agent-harness-architecture-deep-dive-aksahy, aws-bedrock-multi-agent-collaboration-guide
memory/claude-code/obsidian → claude-code-memory-setup-obsidian-graphify, hermes-agent-self-evolving
ai-coding/vscode → claude-code-memory-setup-obsidian-graphify, openclaw-multi-agent-team-practice
astronomy/ai-discovery → minicpm-v-46-13b, karpathy-ai-agent-7-bits-value-decline

Batch results (2026-05-25):

• 4 entities linked: every-ai-subscription→openai-gpt-realtime-voice-models-qbitai, generalization-dynamics→evals-three-methods, anthropic-acquires-stainless→anthropic-12-mcp, build-live-translation→openai-realtime-api-architecture
• Removed non-existent entity links: developer-tools, observability, ai-agent-debugging, opentelemetry, webrtc, stt, tts

Validation command (before committing):

cd /Users/jinguo/wiki && node scripts/wiki-lint.mjs . 2>&1 | grep "^✖"  # should return nothing

Lint false positives — code block examples in documentation

3 remaining lint errors (as of 2026-05-25): all from concepts/wiki-audit-skill.md — patterns [[entities/foo|display]] and [[raw/...]] inside bash/python code blocks that serve as teaching examples. These CANNOT be fixed without either:

• (a) patching scripts/wiki-lint.mjs to exclude code block content from broken link detection
• (b) accepting the false positives as documentation cost

Action: Do NOT try to "fix" these. Accept 3 false-positive errors as documentation overhead. True broken link count = 0.

Node evaluation script (faster than execute_code for large batches):

node -e "
const fs = require('fs');
const data = JSON.parse(fs.readFileSync('/tmp/eval.json'));
data.sort((a,b) => a.score - b.score);
const band09 = data.filter(r => r.score < 10);
// Read body previews
for (const item of band09) {
    const content = fs.readFileSync('entities/' + item.file, 'utf8');
    const fmEnd = content.indexOf('\n---', 4);
    const bodyStart = content.indexOf('\n', fmEnd + 4) + 1;
    item.body_preview = content.slice(bodyStart).trim().slice(0, 1200);
}
fs.writeFileSync('/tmp/batch_preview.json', JSON.stringify(band09, null, 2));
"

Deduplication before commit: Duplicate index entries (same raw article appearing twice) must be removed before committing — the lint duplicate check catches these. Pattern: scan index.md for raw/articles/ slugs appearing more than once, remove the later occurrence.

Mass Broken Raw Citation Cleanup (2026-06-10)

When entities have ^[raw/articles/slug.md] citations where the raw file doesn't exist, lint reports BROKEN CITATION for each occurrence. At scale (598 entities), manual fix is impractical.

Batch fix pattern:

import os, re

raw_dir = '/Users/jinguo/wiki/raw/articles'
raw_set = {f[:-3] for f in os.listdir(raw_dir) if f.endswith('.md')}

fixed = 0
for fn in os.listdir(entities_dir):
    if not fn.endswith('.md'): continue
    fp = os.path.join(entities_dir, fn)
    with open(fp, 'r', errors='replace') as f:
        content = f.read()
    
    # Find all ^[raw/articles/...] citations
    citations = re.findall(r'\^\[raw/articles/([^]|]+?)(?:\.md)?\]', content)
    broken = [c for c in citations if c not in raw_set and c.replace('.md','') not in raw_set]
    
    if not broken: continue
    original = content
    for cite in broken:
        content = content.replace(f"^[raw/articles/{cite}.md]", "")
        content = content.replace(f"^[raw/articles/{cite}]", "")
    
    if content != original:
        with open(fp, 'w', encoding='utf-8') as f:
            f.write(content)
        fixed += 1

Also fix broken frontmatter source references:

# Remove source: [[raw/articles/nonexistent-slug]] lines
content = re.sub(r'source: \[\[raw/articles/nonexistent-slug\]\]\n?', '', content)

Zero-Link Entity Enrichment via Tag+Keyword Matching (2026-06-10)

When hundreds of entities have 0 entity wikilinks, manual linking is impractical. Automated tag+keyword matching works at scale.

Algorithm:

1. Build tag→set(slug) index from all entity frontmatter tags
2. Build slug→title index from frontmatter titles
3. For each zero-link entity: find candidates by tag overlap + title keyword overlap
4. Add ## 相关实体 section with top 3-5 verified wikilinks

Key insight: Use BOTH tag overlap and title keyword overlap — tag-only misses entities with different tag granularity but similar topics; keyword-only is noisy but catches cross-domain connections.

from collections import Counter

# Tag overlap
candidates = Counter()
for t in my_tags:
    for other in tag_index[t]:
        if other != slug: candidates[other] += 1

# Title keyword overlap
title_words = set(re.findall(r'[a-z]{3,}', my_title.lower()))
for other_slug, other_title in slug_title.items():
    if other_slug == slug: continue
    other_words = set(re.findall(r'[a-z]{3,}', other_title.lower()))
    overlap = title_words & other_words
    if len(overlap) >= 2: candidates[other_slug] += len(overlap)

top = candidates.most_common(5)

Result: 921 zero-link entities → 2 (49% → 0%) in two automated batches.

Auto-Tagging Empty tags: Lines (2026-06-10)

Empty tags come in two formats:

1. tags: [] — YAML empty array → match with r'^tags:\s*\[\s*\]'
2. tags: with no value — bare key → match with r'^tags:\s*$'

Both must be handled. The tags: [] regex misses the bare tags: format, which is common in batch-created entities.

Auto-tag rule table (keyword → tags):

tag_rules = {
    'agent': ['agent'], 'harness': ['harness-engineering'], 'skill': ['skill'],
    'claude': ['claude'], 'openai': ['openai'], 'anthropic': ['anthropic'],
    'aws': ['aws'], 'security': ['security'], 'memory': ['memory'],
    'rag': ['rag'], 'multi-agent': ['multi-agent'], 'eval': ['evaluation'],
    'architecture': ['architecture'], 'inference': ['inference'],
    'fine-tun': ['fine-tuning'], 'nvidia': ['nvidia'], 'netflix': ['netflix'],
    # ... extend per vault domain
}

Two-batch results (2026-05-20/21)

• Batch 1 (95 entities, v×c<30 with raw): 48 deleted (non-AI/ML: SAP×4, Trump Media, BTC, Nvidia, Canvas breach, Google Workspace, cPanel, Tether, bluekit, etc.)
• Batch 2 (117 entities, v×c<30): 32 deleted (Gemini App, Google Ads, AWS CDK, EKS Velero, Funnel Builder WooCommerce, OAuth phishing, Tether grants, Twitter thread, green AI, a16z Series C, Python logs, OpenSquilla, npm supply chain, Tencent Hy3, 范凌, 淘宝动效, elasticpp, GPT-5.5 实测翻车, Genkit Middleware, Airbyte, Lovable, Seer Agent, OpenClaw 指南, AGENTS.md 指南, OpenAI Realtime Voice, GRC Now, AWS DevOps 中国区, drinking-llms)

Third batch (2026-05-22, 78 entities v×c<30):

• 44 deleted: product announcements (腾讯员工公寓, kimi-chatbot-release, baichuan-ai-launch), news (vibe-coding to agentic engineering, GPT-5.5 实测), duplicates, non-AI/ML, subjective评测
• 33 kept: substantive AI/ML content from trusted sources (c≥9) — agent-skill-writing-guide, cheriot-ibex-memory-safety, claude-code-governance, gaode-ai-companion-agent-architecture, etc.
• Root cause of historical v×c<30: pipeline score gate (score<49 → skip+delete inbox) was implemented AFTER these were ingested; new ingestions are protected

394 Entity Stubs — Post-Orphan-Raw Batch (2026-05-21)

After classifying 782 orphan raw files (2026-05-21), 394 received minimal stub entities via batch_create_entities.py. These are NOT errors — they are placeholder entities awaiting expansion.

Characteristics:

• Frontmatter + 1-2 bullet points, body < 2KB
• Single → [[raw/articles/slug]] backlink
• No ## 深度分析 section

Priority expansion targets:

• Tag clusters: open-source: 46, aws-china-blog: 51, api: 44, gpt: 34
• Use wiki-entity-expansion skill with subagent workflow

Do NOT confuse with true article-type entities — CVE reports, vendor bulletins, newsletters are correctly thin by design (article_type: true in frontmatter).

c=0 entities need confidence scoring: Many substantive AI/Harness entities (Agent Hooks, AHE, Better-Harness, Claude Code 架构, Project Glasswing, Factory Missions, Harness Engineering 三次进化, Hermes 记忆系统, LLaVA-OneVision-2, Pi Agent, RAG 演进方向, Skills-refiner) have review_value set but confidence=0. These are NOT low quality — they just haven't been scored yet. Fill confidence to normalize scores.

Phase 3 entity batch patching — parallel safe, but sed fix must include entities/

When expanding 3-8 entity files in a single session, patch tool can be called in parallel (different files = independent operations). However, after ALL patches complete, run the sed fix on entities/*.md IN ADDITION to index.md and log.md:

cd <wiki_root> && sed -i '' 's/^|- /- /' entities/*.md && sed -i '' 's/^|- /- /' index.md && sed -i '' 's/^|- /- /' log.md

The |- prefix corruption was observed to affect entities/*.md when the patch tool modifies entity files — not just index.md and log.md as previously documented. Always include entities/*.md in the cleanup pass after batch entity edits.

Bulk concepts/ deletion — classify before deleting (2026-05-20)

Before deleting any concepts/, classify using this script:

import re
from pathlib import Path

concepts_dir = Path("/Users/jinguo/wiki/concepts")
for f in sorted(concepts_dir.glob("*.md")):
    content = f.read_text(errors='replace')
    body = content.split('---', 2)[-1] if '---' in content else content
    wikilinks = re.findall(r'\[\[[^]]+\]\]', body)
    cites = re.findall(r'\[\^[^\]]+\]', body)  # Zettelkasten citations
    paras = [p for p in body.split('\n') if len(p) > 100 and not p.startswith('#') and not p.startswith('-')]
    print(f"{len(wikilinks):3d} links | {len(cites):3d} cites | {len(paras):3d} paras | {f.stat().st_size:6d}b | {f.name}")

Decision rules (2026-05-20 open-sourcing audit):

Type	Size	Content	Action
STUB	<400B	"> Stub page — 内容待补充", 0 wikilinks, 0 cites	DELETE immediately
AI_SUMMARY	3-7KB	No original framework, just summarizes existing entity articles, 0-3 substantive paragraphs	CHECK if referenced → DELETE if orphan, redirect if referenced
REAL_CONTENT	Any	Has original framework definition, citations, proprietary analysis	KEEP

AI_SUMMARY detection signals:

• Page is a compilation of 2-4 existing entity articles with minimal original text
• No proprietary framework terminology or coined terms
• Paragraphs are close paraphrasing of source articles
• Content could be regenerated by reading the cited entities

REAL_CONTENT signals:

• Author's own framework terminology (first-use of a coined term)
• Direct evidence of original reasoning (not just paraphrasing)
• Proprietary data, benchmarks, or case studies not in any single source
• The concept defines something that doesn't exist in any entity page

sha256 block trapped outside frontmatter — causes MISSING sha256 lint error

When entity files have sha256: in the body section (between body and --- separator), lint scans only the frontmatter block and reports MISSING sha256. Detection and fix — see wiki-audit-and-repair skill Pattern 4.

Truncated slug entities — problem was exaggerated (2026-05-20)

Systematic scan found 0 entity files with slug ≥200 chars. The previously-reported "12 truncated slug duplicates" were different articles sharing a long prefix (e.g., harness-engineering prefix of 4 different articles). No OS-level filename truncation exists in the current corpus. Real deduplication needs are limited to:

1. Tether file variants (微信 title truncation artifacts — multiple slugs for same article)
2. True content duplicates (compare body sha256)

Wiki-lint frontmatter regex requires blank-line separation — silent failure mode

Pattern: Entity has frontmatter that looks valid in YAML but wiki-lint.mjs reports NO FRONTMATTER.

Root cause: Lint uses /^---\n([\s\S]*?)\n---/ — requires --- on its own line AND closing --- on its own line with a trailing newline. If frontmatter ends with sources:\n - url\n# Heading (no blank line between last field and closing ---), the regex fails silently.

Detection:

import re
fpath = '/Users/jinguo/wiki/entities/SOME_FILE.md'
content = open(fpath).read()
fm_match = re.match(r'^---\n([\s\S]*?)\n---', content)
print('Lint-compatible:', fm_match is not None)

Fix: Ensure frontmatter ends with a blank line before ---:

sources:
  - raw/articles/some-article

---

# Actual heading

Also: fields like confidence: instead of review_confidence:, or source: instead of inside sources:, do NOT cause NO FRONTMATTER errors — they cause MISSING type or other warnings. Only the missing closing --- triggers NO FRONTMATTER.

wiki-lint.mjs EXCESS INFERRED = warnings only, exit code stays 0

When expanding entities, wiki-lint.mjs may report EXCESS INFERRED: entities/X has N/M uncited paragraphs (P%) for the newly expanded files. These are warnings, not errors — the lint exits with code 0 and does not block completion. The warnings appear because newly written synthesis paragraphs lack ^[raw/articles/slug.md:LINE_RANGE] citation markers, which is expected for LLM-generated analysis sections.

Action: After running lint, filter for actual errors separately:

node scripts/wiki-lint.mjs . 2>&1 | grep "^✖"           # actual errors
node scripts/wiki-lint.mjs . 2>&1 | grep "EXCESS"      # warnings only

If the only lint output is EXCESS INFERRED warnings, the session is clean — proceed to closeout without further fixes.

EXCESS INFERRED is a FALSE POSITIVE for wikis using wikilinks for provenance

2026-05-24 finding: A-grade wiki (no errors, 0 true orphans, 0 YAML/tag/title issues) still shows 1490 EXCESS INFERRED entities — and this is misleading.

Root cause: wiki-lint.mjs counts ^[raw/articles/slug.md] (Zettelkasten/Obsidian citation plugin format) as the ONLY provenance citation format. But this wiki uses [[raw/articles/slug]] wikilinks for provenance — which are tracked separately as wikilinks, NOT as citations in the lint's sense.

Evidence (1552 entities):

• entity→raw wikilinks: 1844 (real provenance via [[raw/...]])
• entity→entity wikilinks: 4629 (cross-references)
• ^[raw/articles/...] citations: 0 (unused citation format)
• Total uncited paragraphs: 53,504/54,941 (97.4%) — but this counts paragraphs lacking ^[...] markers, ignoring the [[raw/...]] wikilinks that DO exist in these bodies

Interpretation:

Scenario	Entity count	Meaning
Has raw + [[raw/...]] wikilinks + 0 ^[...]	1252	Legitimate — wikilinks ARE provenance, lint ignores them
No raw + 100% uncited	238	Synthesized from summary/notes, not from raw articles — legitimate for concept/synthesis pages
No raw + high uncited %	~165	Low-priority synthesized content (adversarial-verification, agentcore-harness, etc.)

What to actually measure for wiki quality (not paragraph-level citation coverage):

• Broken wikilinks = 0 ✓ (already clean)
• True orphans = 0 ✓ (already clean)
• YAML/empty-tags/missing-title errors = 0 ✓ (already clean)
• Entity coverage of raw articles = 1252/1552 with some raw link (already high)

Do NOT try to "fix" EXCESS INFERRED by adding ^[raw/articles/...] citations — it would require ~53,000 paragraph-level modifications and the resulting format would be inconsistent with the existing wikilink-based style. EXCESS INFERRED warnings are informational only and do not indicate actual quality problems in a wikilink-based wiki.

The real quality signal: A wiki with 0 broken links, 0 true orphans, 0 metadata errors, and good cross-referencing (4629 entity→entity wikilinks) is high-quality regardless of ^[...] citation count.

Diagnostic reference: Full diagnostic commands and truth table of what metrics actually mean — see wiki-audit skill's references/wiki-quality-diagnostic.md.

Bulk paragraph spacing normalization — AI-generated content dumps

Many entities synthesized from AI output have 0% blank lines — continuous prose blocks that are unreadable. Fix: inject blank lines algorithmically across all affected entities.

Detection: execute_code counts sum(1 for l in body.split('\n') if not l.strip()) = 0 or near-zero.

Spacing algorithm:

import re, os

def add_spacing(body):
    lines = body.split('\n')
    result = []
    prev_type = None
    
    for line in lines:
        stripped = line.strip()
        if not stripped:       curr_type = 'blank'
        elif stripped.startswith('```'): curr_type = 'code'
        elif stripped.startswith('##'): curr_type = 'h2'
        elif stripped.startswith('#'):   curr_type = 'h3'
        elif re.match(r'^[-*+]\s', stripped): curr_type = 'list'
        else:                          curr_type = 'prose'
        
        # Blank before list following prose
        if curr_type == 'list' and prev_type in ('prose', 'code', 'h3') and result and result[-1].strip():
            result.append('')
        # Blank before h2/h3 following non-blank non-heading
        if curr_type in ('h2', 'h3') and prev_type not in (None, 'blank', 'h2', 'h3'):
            if result and result[-1].strip():
                result.append('')
        
        result.append(line)
        prev_type = curr_type
    
    # Collapse multiple blanks to max 2
    final = []
    blank_run = 0
    for line in result:
        if not line.strip():
            blank_run += 1
            if blank_run <= 2:
                final.append(line)
        else:
            blank_run = 0
            final.append(line)
    return '\n'.join(final)

# Apply to all entities, preserve frontmatter
entities_dir = '/Users/jinguo/wiki/entities'
for fname in sorted(os.listdir(entities_dir)):
    if not fname.endswith('.md'): continue
    fpath = os.path.join(entities_dir, fname)
    content = open(fpath, encoding='utf-8').read()
    fm_match = re.match(r'^---\n[\s\S]*?\n---\n', content)
    if fm_match:
        fm, body = fm_match.group(0), content[fm_match.end():]
    else:
        fm, body = '', content
    new_body = add_spacing(body)
    if new_body != body:
        open(fpath, 'w', encoding='utf-8').write(fm + new_body)

Results (2026-05-23):

• 1249 entities modified (+67 blank lines avg)
• 302 entities skipped (already had spacing)
• 0 errors, lint stays clean