wiki-evolver

name	wiki-evolver
description	Evolve a mature Markdown wiki into a research and practice engine. Use for frontier mapping, paper/practice backlog generation, dashboard updates, and repeated knowledge-vault improvement loops on top of llm-wiki.
version	1.0.0
author	Hermes Agent
license	MIT
metadata	{"hermes":{"tags":["wiki","knowledge-system","research","orchestration","synthesis"],"category":"research","related_skills":["wiki-pipeline","llm-wiki","web-content-reviewer","obsidian"]}}

Wiki Evolver

Run the layer above ingest. This skill turns an already-usable Markdown wiki into a compounding system that keeps producing stronger questions, better navigation, practice artifacts, and improved skills.

Activation

Use this skill when the user asks to:

• improve or evolve an existing wiki rather than ingest a single source
• discover the most valuable next questions inside a vault
• turn current vault material into frontier maps, paper candidates, practice backlogs, or dashboard updates
• repeatedly run a knowledge-evolution cycle over a mature Obsidian/Markdown knowledge base
• refine the knowledge system itself, not just add one more note

Operating Contract

1. wiki-evolver orchestrates; it does not replace llm-wiki.
2. Any durable wiki write must still follow llm-wiki completion gates.
3. Any source scoring or URL gatekeeping still belongs to web-content-reviewer.
4. Every cycle must produce at least one durable outcome, not just analysis.
5. A cycle is complete only when the chosen artifact is updated, navigation is current, and validation is reported.

Workflow Decision Table

Situation	Primary output	Reference
Vault is content-rich but direction-poor	frontier map	references/evolution-loop.md
Strong clusters exist and can become external artifacts	paper backlog	references/artifact-decision-matrix.md
Theory is ahead of execution	practice backlog or playbook	references/artifact-decision-matrix.md
The system needs better visibility and cadence	dashboard update	references/evolution-loop.md
The process itself is drifting	Skill/template refinement	references/execution-template.md

Core Loop

Use this sequence every time:

Orient -> Diagnose -> Choose leverage track -> Produce artifact -> Govern -> Close out

• Orient: read SCHEMA.md, index.md, recent log.md, and the strongest query/navigation pages.
• Diagnose: find where the vault is currently bottlenecked: missing question layer, missing artifact layer, weak navigation, or weak workflow.
• Choose leverage track: pick exactly one primary track for the cycle: frontier, paper, practice, dashboard, or skill.
• Produce artifact: create or update the durable page/template/checklist that best relieves the bottleneck.
• Govern: update index.md, log.md, and adjacent navigation when the wiki changes.
• Close out: report what changed, why it matters, and what the next best cycle should target.

Durable Outcomes

At least one of these must happen in a successful cycle:

• a new or improved frontier map
• a new or improved paper backlog
• a new or improved engineering practice backlog
• a new or improved dashboard / topic map / learning path
• a Skill, checklist, template, or validator refinement
• a structural repair that materially improves future evolution work — this includes: deleting weak/empty pages (v×c < 30 entities, < 3KB concepts), fixing wikilink orphans, consolidating duplicate pages, or adding concept pages for undersynthesized tag clusters. Governance fixes compound because they raise the baseline quality of every future query.

Core Rules

• Prefer high-leverage orchestration over low-value page churn.
• Queries must be real research questions, not topic maps. A query page's H1 must end with ? or ？ and contain evidence synthesis — not a navigation list of [[entities/...]] links. See references/evolution-loop.md → Query Taxonomy for the three-type classification and conversion decision tree.
• Do not create query pages that merely restate a conversation.
• Prefer updating existing control pages before creating near-duplicate ones.
• Convert recurring reasoning into templates, checklists, or validators.
• Keep the main Skill lean; put concrete execution detail in references/.
• When a wiki change is required, load and follow llm-wiki.
• When the cycle depends on scored external sources, run web-content-reviewer first.

Completion Gate

Report success only when all are true:

1. The cycle chose a clear leverage track.
2. At least one durable outcome was created or improved.
3. Any wiki changes followed llm-wiki rules, including index/log handling.
4. Dashboard/navigation impact was considered and updated when useful.
5. Validation passed via node scripts/wiki-lint.mjs when wiki files changed (explicit explanation when no wiki write occurred).

If the cycle produced only observations, report it as incomplete.

Failure Semantics

• "I found interesting themes" is not completion.
• "The vault is promising" is not completion.
• If a chosen artifact was not updated, the cycle failed.
• If wiki pages changed but index/log or validation was skipped, the cycle failed.
• If no durable outcome is justified, say so explicitly and stop instead of fabricating one.

Bidirectional Link Quality — Two Axes

Query and concept pages have two link axes that must both be healthy:

Axis 1: Out-links (Page → Entity/Concept)

What it measures: How well a page is anchored to the wiki's knowledge base.

Metric: Each query should link to 3+ [[entities/...]] and [[concepts/...]] pages.

Fix thin out-links:

1. Read the page content
2. Search wiki for relevant entities/concepts: search_files(pattern="relevant-term", path="entities")
3. Add 3-8 wikilinks in a ## 相关实体 / ## 相关概念 section

Axis 2: In-links (Entity/Concept → Page)

What it measures: How discoverable the page is from the rest of the wiki.

Metric: Each page should be referenced by at least 1 other page.

Fix orphan pages (0 in-links):

1. Read the orphan page to understand its topic
2. Find 1-2 relevant concept or entity pages
3. Add [[queries/page-name]] or [[concepts/page-name]] reference in those pages

Bidirectional Health Metrics

Directory	Out-links avg	In-links avg	Orphans
concepts/	≥15	≥12	0
queries/	≥10	≥4	0
entities/	≥5	≥3	<50

Run after any expansion batch:

node scripts/wiki-lint.mjs . 2>&1 | grep -A 10 "Per-directory"

Governance Technique: Mass Broken Citation Cleanup

When entities contain ^[raw/articles/...] citations referencing raw files that don't exist on disk, these broken citations silently degrade wiki quality. A bulk cleanup pass found 598 such citations in one session.

Detection

import os, re

wiki = "/Users/jinguo/wiki"
entities_dir = f"{wiki}/entities"
raw_dir = f"{wiki}/raw/articles"
raw_set = {f[:-3] for f in os.listdir(raw_dir) if f.endswith('.md')}

broken_count = 0
for fn in os.listdir(entities_dir):
    if not fn.endswith('.md'): continue
    fp = os.path.join(entities_dir, fn)
    with open(fp, 'r', encoding='utf-8', errors='replace') as f:
        content = f.read()
    citations = re.findall(r'\^\[raw/articles/([^]|]+?)(?:\.md)?\]', content)
    broken = [c for c in citations if c not in raw_set and c.replace('.md', '') not in raw_set]
    broken_count += len(broken)

print(f"Total broken raw citations: {broken_count}")

Fix — remove broken citation markers

for fn in os.listdir(entities_dir):
    if not fn.endswith('.md'): continue
    fp = os.path.join(entities_dir, fn)
    with open(fp, 'r', encoding='utf-8', errors='replace') as f:
        content = f.read()
    citations = re.findall(r'\^\[raw/articles/([^]|]+?)(?:\.md)?\]', content)
    broken = [c for c in citations if c not in raw_set and c.replace('.md', '') not in raw_set]
    if broken:
        original = content
        for cite in broken:
            content = content.replace(f"^[raw/articles/{cite}.md]", "")
            content = content.replace(f"^[raw/articles/{cite}]", "")
        if content != original:
            with open(fp, 'w', encoding='utf-8') as f:
                f.write(content)

Piped wikilink fix

When fixing broken [[entities/slug]] or [[concepts/slug]] links that the lint reports, check for BOTH bare and piped forms:

import re
# Bare: [[entities/slug]]
content = content.replace("[[entities/slug]]", "**Display Text**")
# Piped: [[entities/slug|Display Text]]
content = re.sub(r'\[\[entities/slug\|[^\]]+\]\]', '**Display Text**', content)

Simple .replace() misses piped forms because the display text varies.

Prevention

When adding ^[raw/articles/...] citations during expansion, ALWAYS verify the raw slug exists in raw_set first. For entities without a matching raw article, use entity wikilinks or plain text references — never fabricate raw citations.

Governance Technique: Mass Wikilink Deduplication

When wikilink duplication is found (same [[entity]] appearing 2+ times in one file, or the same ^[raw/articles/...] footnote repeated dozens of times per file across hundreds of files), use this detection → fix → validate pattern.

Detection

import re, os
from collections import Counter

entities_dir = '/Users/jinguo/wiki/entities'

for fname in os.listdir(entities_dir):
    if not fname.endswith('.md'):
        continue
    fpath = os.path.join(entities_dir, fname)
    slug = fname[:-3]
    with open(fpath) as f:
        content = f.read()

    wikilinks = re.findall(r'\[\[([^\]|]+?)(?:\|[^\]]+)?\]\]', content)

    # 1. True self-links (entity → itself): 0 tolerance, always fix
    for link in wikilinks:
        if '/' not in link and link == slug:
            print(f"SELF: {fname}: [[{link}]]")

    # 2. Duplicate entity/concept wikilinks per file
    entity_links = [w.strip() for w in wikilinks if not w.strip().startswith('raw/')]
    link_counts = Counter(entity_links)
    dups = {l: c for l, c in link_counts.items() if c > 1}

    # 3. Duplicate footnote refs (^[raw/articles/...]) — biggest噪音 source
    footnotes = re.findall(r'\^\[([^\]]+)\]', content)
    fn_counts = Counter(footnotes)
    dups = {f: c for f, c in fn_counts.items() if c > 1 and f.startswith('raw/')}

Fix — keep first occurrence, remove duplicates

# Wikilinks: keep first, remove rest (reverse order to preserve offsets)
matches = list(re.finditer(pattern, content))
for m in reversed(matches[1:]):
    content = content[:m.start()] + '___REMOVED___' + content[m.end():]
content = content.replace('___REMOVED___', '')
content = re.sub(r'\n{3,}', '\n\n', content)

# Footnotes: remove marker only, keep text content
line = re.sub(r'\^\[' + re.escape(fn) + r'\]', '', line)

Severity guide

Type	Tolerance	Action
True self-link (entity → itself)	0	Fix immediately
Duplicate ^[raw/articles/...]	High (40k+ typical in large vaults)	Batch fix — keep 1 per source
Duplicate entity/concept wikilink	Medium (300+ typical)	Batch fix — keep first
raw/article wikilinks in footnotes	Normal	Leave — correct usage

Post-fix validation

node scripts/wiki-lint.mjs . 2>&1 | grep "INDEX DRIFT\|ERROR"
sed -i '' 's/^|- /- /' index.md log.md
git add entities/ index.md log.md && git commit -m "Deduplicate wikilinks and footnotes across entities"

Mass Concept Connection Application (2026-06-10)

When 70+ concepts need the 关联实体 (upstream/downstream/parallel) pattern from references/entity-connection-pattern.md, manual application per concept is impractical. An automated approach works:

Algorithm:

1. Read each concept's existing [[entities/...]] wikilinks
2. Split links into thirds: first 1/3 = upstream, middle 1/3 = downstream, last 1/3 = parallel
3. Insert ## 关联实体 section with structured categories before any → [[raw/...]] or final sections
4. Skip concepts that already have the pattern

for fn in sorted(os.listdir(concepts_dir)):
    # ... read concept ...
    if '## 关联实体' in c and ('上游' in c or '下游' in c): continue  # Already done
    
    entity_links = re.findall(r'\[\[entities/([^]|]+)', c)
    if len(entity_links) < 3: continue
    
    n = len(entity_links)
    up = entity_links[:n//3]
    down = entity_links[n//3:2*n//3]
    parallel = entity_links[2*n//3:]
    
    section = "\n## 关联实体\n"
    if up:
        section += "\n**上游依赖**:\n"
        for l in up[:3]: section += f"- [[entities/{l}]] — 提供基础理论/方法\n"
    # ... same for down and parallel ...

Result: 72/74 concepts enriched in one automated pass. The remaining 2 had <3 entity links and needed manual linking first.

Note: The 1/3 split is a heuristic. For concepts with strong domain knowledge, manual categorization is better. But at scale, the heuristic produces useful connections that are better than no structure.

Extended Evolution Phases (2026-06-10)

These phases run after the core loop has produced a mature vault (1000+ entities, concepts, queries). They address the next tier of quality gaps.

Phase 5: Strategic Artifacts (Navigation and Action)

Generate wiki-level strategic documents that synthesize cross-entity patterns into actionable output. See references/strategic-artifact-generation-2026-06-10.md for full patterns and script templates.

Dashboard (wiki root Dashboard.md): Navigation hub with 7 sections -- total counts, top tags, top entities by in-links, concept list, query list, size distribution, quality metrics. Regenerate after any structural change.

Research Frontier Map (queries/research-frontier-map.md): Cluster frontier questions from tag co-occurrence. 6 clusters typical (derived from top tags). Each cluster: description, key entities, 3-4 open questions.

Engineering Practice Backlog (queries/engineering-practice-backlog.md): Convert strongest theoretical areas into concrete playbooks/checklists/templates. Link to source entities for traceability.

Phase 6: Entity Stub Ingestion (Coverage First)

When raw articles lack corresponding entities (orphan raws), batch-create minimal stubs. Priority: create stubs for ALL orphan raws first (maximize coverage), then expand highest-value ones in subsequent sessions with 深度分析.

Algorithm per orphan raw: (1) Read raw, extract H1 title, (2) Extract first non-empty paragraph as summary, (3) Auto-infer tags from title/slug keywords, (4) Write minimal entity frontmatter + summary + raw link. Proven at 292 stubs in one batch.

Phase 7: Concept Relationship Enrichment

Add 关联实体 section to ALL concepts using structured pattern: 上游依赖, 下游应用, 平行协作. Makes concepts proper navigational hubs, not just definition pages. Proven at 72/74 concepts in one automated pass (see Mass Concept Connection section above).

Phase 8: Full Stub Expansion (100% 深度分析 Coverage)

After Phase 6 stub creation, expand ALL remaining stubs (those with raw articles) to achieve 100% 深度分析 coverage. Use bulk automated expansion via execute_code rather than subagents — proven at 302 stubs in one pass (~1 second). See wiki-entity-expansion skill → "Bulk Automated Expansion" section and references/bulk-expansion-2026-06-10.md for the complete algorithm.

Key techniques:

• extract_meaningful_text() — strips HTML/CSS/noise from raw articles
• extract_key_sentences() — filters metadata/opening lines, keeps substantive content
• Tag inference from content keywords — replaces 'uncategorized' with domain tags
• Related entity discovery by tag overlap (≥2 shared tags = related, deterministic)
• Template-based ## 深度分析 + ## 实践启示 generation

Post-expansion cleanup (mandatory):

1. sed -i '' 's/^|- /- /' entities/*.md — fix YAML list rendering
2. Zero-link scan and fix (entities with no [[entities/...]] wikilinks)
3. type: entity frontmatter mass-fix for entities missing this field
4. Raw article frontmatter mass-fix (NO FRONTMATTER lint errors)
5. Index update (add missing entities/queries/raws to index.md)
6. Lint-verify-commit

Quality note: Bulk expansion produces consistent template structure. Next phase should refine with raw-content-specific analysis. Create queries/review-queue.md to track entities needing refinement.

Proven results (2026-06-10): 302 stubs expanded, 深度分析 86%→100%, zero-link 10→0, cross-refs 10,447→12,766, 303 entities fixed with type: entity, 12 raws fixed with frontmatter.

Progressive References

Load only what the current cycle needs:

• references/evolution-loop.md — how to run frontier -> paper -> practice -> dashboard cycles.
• references/query-taxonomy.md — three-type classification for queries/ files (real question vs. synthesis doc vs. topic map), conversion patterns, and post-cleanup verification.
• references/artifact-decision-matrix.md — choose the right durable artifact for the current bottleneck.
• references/execution-template.md — reusable execution checklist and session template.
• references/closeout-template.md — required success/failure closeout for evolution cycles.
• references/entity-connection-pattern.md — systematic methodology for connecting wiki entities using upstream/downstream/parallel taxonomy and application scenarios.
• references/strategic-artifact-generation-2026-06-10.md — Dashboard, Frontier Map, and Practice Backlog generation patterns with script templates and proven cluster derivation.
• references/session-2026-06-10-comprehensive-evolution.md — full vault quality uplift session: mass broken citation cleanup, zero-link entity enrichment, orphan reduction, concept connection pattern at scale.
• references/9-step-paradigm-alignment.md — load when the user mentions Karpathy / 超级猛 / hermes-wiki / 9 步法 / 7 维诊断 / 节点类型分离 / "演化空间". Full playbook: 7-dim diagnosis framework, P0→P1→P2 priority order, MOC migration recipe, wiki-lint upgrade for new node types, drafts 100%-internal-source rule, concepts stub-then-grow at scale, comparison templates, lint self-bite pitfalls (示例 wikilink 当真链), cron race V2 handling, delegate_task quota limits, final report format.