Menu
Karpathy's LLM Wiki: build/query interlinked markdown KB.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Comprehensive wiki maintenance: daily structural health checks (index reconciliation, log separator fixes, pipeline watchdog alerts), comparison page updates (adding items to multi-section comparison tables), and page relocation (moving/renaming pages while maintaining link integrity).
Pre-flight checklist and procedures for archiving, deleting, or migrating Hermes skills. Prevents accidental removal of cron-referenced skills. Covers the 3-layer skill structure and config.yaml management. Includes skill inventory management, promotion workflows, and archival conventions.
Deep analysis of blog authors' recent thoughts, philosophy, and positions. Goes beyond entity page creation to extract cited ideological positions, track thought evolution, and enable ongoing RSS monitoring for thought updates.
Query the blogwatcher-cli SQLite database for RSS scan results. Use pre-verified column names and query templates to avoid errors.
Synthesize perspectives from multiple opinion leaders on a shared topic into a cross-referenced wiki index page. Use when the user wants to consolidate different viewpoints on the same concept.
Daily RSS scan triage workflow โ scan blogs, triage articles, ingest into wiki, commit changes
| name | llm-wiki |
| description | Karpathy's LLM Wiki: build/query interlinked markdown KB. |
| version | 2.1.0 |
| author | Hermes Agent |
| license | MIT |
| platforms | ["linux","macos","windows"] |
| metadata | {"hermes":{"tags":["wiki","knowledge-base","research","notes","markdown","rag-alternative"],"category":"research","related_skills":["obsidian","arxiv"]}} |
Build and maintain a persistent, compounding knowledge base as interlinked markdown files. Based on Andrej Karpathy's LLM Wiki pattern.
Unlike traditional RAG (which rediscovers knowledge from scratch per query), the wiki compiles knowledge once and keeps it current. Cross-references are already there. Contradictions have already been flagged. Synthesis reflects everything ingested.
Division of labor: The human curates sources and directs analysis. The agent summarizes, cross-references, files, and maintains consistency.
Use this skill when the user:
Location: Set via WIKI_PATH environment variable (e.g. in ~/.hermes/.env).
If unset, defaults to ~/wiki.
WIKI="${WIKI_PATH:-$HOME/wiki}"
In this environment ~/wiki should resolve to ~/ai-topics/wiki. Do not silently
switch to any alternate location.
The wiki is just a directory of markdown files โ open it in Obsidian, VS Code, or any editor. No database, no special tooling required.
wiki/
โโโ SCHEMA.md # Conventions, structure rules, domain config
โโโ index.md # Sectioned content catalog with one-line summaries
โโโ log.md # Chronological action log (append-only, rotated yearly)
โโโ raw/ # Layer 1: Immutable source material
โ โโโ articles/ # Web articles, clippings
โ โโโ papers/ # PDFs, arxiv papers
โ โโโ transcripts/ # Meeting notes, interviews
โ โโโ assets/ # Images, diagrams referenced by sources
โโโ entities/ # Layer 2: Entity pages (people, orgs, products, models)
โโโ concepts/ # Layer 2: Concept/topic pages
โโโ comparisons/ # Layer 2: Side-by-side analyses
โโโ queries/ # Layer 2: Filed query results worth keeping
โโโ events/ # Layer 2: Time-bounded events (launches, incidents, announcements)
Layer 1 โ Raw Sources: Immutable. The agent reads but never modifies these. Layer 2 โ The Wiki: Agent-owned markdown files. Created, updated, and cross-referenced by the agent.
When a topic grows beyond 5-10 concept pages, organize into a subdirectory with a synthesis index:
wiki/concepts/
โโโ agentic-engineering.md # Parent overview page (brief)
โโโ agentic-engineering/ # Subdirectory for deep dives
โ โโโ _index.md # Cross-leader synthesis hub
โ โโโ red-green-tdd.md
โ โโโ first-run-the-tests.md
โ โโโ using-git-with-agents.md
โโโ other-concept.md
The _index.md synthesizes perspectives from multiple opinion leaders on the shared topic,
mapping convergences, divergences, and unresolved questions. Use the cross-leader-synthesis
skill for this pattern.
Rules:
_index.md is the entry point โ always read it first for a topic overviewagentic-engineering.md) stays as a brief definition + links to subdirectory_index.md_index.md links to each leader's entity page via [[wikilinks]]
Layer 3 โ The Schema: SCHEMA.md defines structure, conventions, and tag taxonomy.When the user has an existing wiki, always orient yourself before doing anything:
โ Read SCHEMA.md โ understand the domain, conventions, and tag taxonomy.
โก Read index.md โ learn what pages exist and their summaries.
โข Scan recent log.md โ read the last 20-30 entries to understand recent activity.
WIKI="${WIKI_PATH:-$HOME/wiki}"
# Orientation reads at session start
read_file "$WIKI/SCHEMA.md"
read_file "$WIKI/index.md"
read_file "$WIKI/log.md" offset=<last 30 lines>
Only after orientation should you ingest, query, or lint. This prevents:
For large wikis (100+ pages), also run a quick search_files for the topic
at hand before creating anything new.
When the user asks to create or start a wiki:
$WIKI_PATH env var, or ask the user; default ~/wiki)SCHEMA.md customized to the domain (see template below)index.md with sectioned headerlog.md with creation entryAdapt to the user's domain. The schema constrains agent behavior and ensures consistency:
# Wiki Schema
## Domain
[What this wiki covers โ e.g., "AI/ML research", "personal health", "startup intelligence"]
## Conventions
- File names: lowercase, hyphens, no spaces (e.g., `transformer-architecture.md`)
- Every wiki page starts with YAML frontmatter (see below)
- Use `[[wikilinks]]` to link between pages (minimum 2 outbound links per page)
- When updating a page, always bump the `updated` date
- Every new page must be added to `index.md` under the correct section
- Every action must be appended to `log.md`
- **Provenance markers:** On pages that synthesize 3+ sources, append `^[raw/articles/source-file.md]`
at the end of paragraphs whose claims come from a specific source. This lets a reader trace each
claim back without re-reading the whole raw file. Optional on single-source pages where the
`sources:` frontmatter is enough.
## Frontmatter
```yaml
---
title: Page Title
created: YYYY-MM-DD
updated: YYYY-MM-DD
type: entity | concept | comparison | query | summary
tags: [from taxonomy below]
sources: [raw/articles/source-name.md]
# Optional quality signals:
confidence: high | medium | low # how well-supported the claims are
contested: true # set when the page has unresolved contradictions
contradictions: [other-page-slug] # pages this one conflicts with
---
Required fields: title, created, updated, type, tags, sources. Omitting sources is the most common frontmatter gap at scale โ if you have no source to cite, use sources: [].
confidence and contested are optional but recommended for opinion-heavy or fast-moving
topics. Lint surfaces contested: true and confidence: low pages for review so weak claims
don't silently harden into accepted wiki fact.
Raw sources ALSO get a small frontmatter block so re-ingests can detect drift:
---
source_url: https://example.com/article # original URL, if applicable
ingested: YYYY-MM-DD
sha256: <hex digest of the raw content below the frontmatter>
---
The sha256: lets a future re-ingest of the same URL skip processing when content is unchanged,
and flag drift when it has changed. Compute over the body only (everything after the closing
---), not the frontmatter itself.
[Define 10-20 top-level tags for the domain. Add new tags here BEFORE using them.]
Example for AI/ML:
Rule: every tag on a page must appear in this taxonomy. If a new tag is needed, add it here first, then use it. This prevents tag sprawl.
_archive/, remove from indexOne page per notable entity. Include:
One page per concept or topic. Include:
Side-by-side analyses. Include:
When new information conflicts with existing content:
contradictions: [page-name]
### index.md Template
The index is sectioned by type. Each entry is one line: wikilink + summary.
```markdown
# Wiki Index
> Content catalog. Every wiki page listed under its type with a one-line summary.
> Read this first to find relevant pages for any query.
> Last updated: YYYY-MM-DD | Total pages: N
## Entities
<!-- Alphabetical within section -->
## Concepts
## Comparisons
## Queries
## Events
Events section: Time-bounded happenings (product launches, security incidents, funding announcements). Use type: event in frontmatter. These are distinct from concept pages which capture enduring knowledge.
Scaling rule: When any section exceeds 50 entries, split it into sub-sections
by first letter or sub-domain. When the index exceeds 200 entries total, create
a _meta/topic-map.md that groups pages by theme for faster navigation.
# Wiki Log
> Chronological record of all wiki actions. Append-only.
> Format: `## [YYYY-MM-DD] action | subject`
> Actions: ingest, update, query, lint, create, translate, archive, delete
> When this file exceeds 500 entries, rotate: rename to log-YYYY.md, start fresh.
## [YYYY-MM-DD] create | Wiki initialized
- Domain: [domain]
- Structure created with SCHEMA.md, index.md, log.md
Log rotation: When log.md exceeds 500 lines, archive it as log-YYYY.md and create a fresh file with the header + rotation entry. See wiki-graph-health skill's references/log-rotation.md for the full procedure.
When the user provides a source (URL, file, paste), integrate it into the wiki:
โ Capture the raw source:
web_extract to get markdown, save to raw/articles/web_extract (handles PDFs), save to raw/papers/raw/ subdirectoryraw/articles/karpathy-llm-wiki-2026.mdsource_url, ingested, sha256 of the body).
On re-ingest of the same URL: recompute the sha256, compare to the stored value โ
skip if identical, flag drift and update if different. This is cheap enough to
do on every re-ingest and catches silent source changes.references/single-page-research-collections.md for full procedure.โก Discuss takeaways with the user โ what's interesting, what matters for the domain. (Skip this in automated/cron contexts โ proceed directly.)
โข Check what already exists โ search index.md and use search_files to find
existing pages for mentioned entities/concepts. This is the difference between
a growing wiki and a pile of duplicates.
โฃ Write or update wiki pages:
updated date.
When new info contradicts existing content, follow the Update Policy.sources: frontmatter field. For raw article ingestion: sources: [raw/articles/<filename>]. For web extraction: list the URLs. If unsure, use sources: [] โ never omit the field entirely.[[wikilinks]]. Check that existing pages link back.wiki/SCHEMA.md and ensure ALL tags come from its taxonomy. If you need a new tag, add it to SCHEMA.md FIRST. Never use ad-hoc or composite kebab-case tags. Violations are blocked by pre-commit hook.^[raw/articles/source.md]
markers to paragraphs whose claims trace to a specific source.confidence: medium or low in frontmatter. Don't mark high unless the
claim is well-supported across multiple sources.โค Update navigation:
index.md under the correct section, alphabeticallylog.md: ## [YYYY-MM-DD] ingest | Source Titleโฅ Report what changed โ list every file created or updated to the user.
A single source can trigger updates across 5-15 wiki pages. This is normal and desired โ it's the compounding effect.
When the wiki has Japanese-language content that needs systematic English translation,
run the batch workflow documented in references/jp-en-translation-batch.md:
Scan and rank โ walk all non-raw .md files, count JP chars in body (after frontmatter)
Select top N โ recommend 8 files per batch (~30K JP chars). Exclude log.md/log-2026.md.
Choose translation method by file type โ Three translation approaches, each suited to a different file structure:
| File Type | Best Approach | When It Works | When It Fails |
|---|---|---|---|
| Fully-JP concept/entity (<200 lines, >90% JP in body) | Full body rewrite via write_file or execute_code (Python triple-quoted strings) | Coherent JP paragraphs that translate cleanly to one block | Multi-section files where EN/JP boundaries are ambiguous |
| Scattered-JP (mostly EN, JP in 10+ locations: headers, table cells, inline text) | delegate_task subagent with context-aware translation instructions | Files with scattered JP where context matters for accurate translation (log files, index files) | Fully-JP files >500 lines where subagent hits token limits |
| Light scattered-JP (JP in <10 specific locations) | Per-file str.replace() dict via execute_code | Known replacement strings in fixed positions (section headers, table column headers) | Files where replacement strings have whitespace mismatches or Unicode-variant characters |
โ ๏ธ Chinese vs Japanese CJK characters: When scanning for Japanese content, the regex [\u4E00-\u9FFF] captures both Chinese characters (ๆผขๅญ) and Japanese kanji. Many wiki files contain Chinese proper nouns (person names like ๅง้กบ้จ, organization names like ๆบ่ฐฑAI, ๆไนๆ้ข, product names). These are NOT Japanese and should be preserved. Use the hiragana/katakana-only regex [\u3040-\u309F\u30A0-\u30FF] to identify translatable Japanese text. See references/chinese-proper-noun-handling.md for the verification script and detailed handling procedure. When a scan reports "CJK characters," verify whether they're Chinese proper nouns before attempting translation.
โ ๏ธ str.replace() has fundamental limits for scattered JP โ A replacement dict with 200+ entries may only remove 15-30% of JP chars per pass on log files with scattered JP. The silent non-match rate is high because JP particles (ใฎ, ใ, ใธ) appear in many different contexts. After 3 aggressive passes with 400+ total replacement entries, ~60% of scattered JP remained. Delegate_task subagents are dramatically more effective for this case โ each subagent can understand context and properly translate scattered JP in a single pass. Tested on a 1732-line log file with 4,015 JP chars: str.replace() removed 670 over 3 passes; one subagent pass removed the remaining 3,345 with context-aware translation.
When to use which approach in a batch: Split the 8 files into groups:
execute_code with full body rewritesdelegate_task (1 subagent per file, up to 3 in parallel)execute_code (full rewrite), 3 log files via delegate_task (scattered-JP translation). All 8 achieved 0 residual JP.โ ๏ธ Fullwidth colon trap: After translation, 6 remaining "JP chars" in a 1732-line file turned out to be fullwidth colons ๏ผ (U+FF1A) โ CJK punctuation, not Japanese language content. These appear in log entries, table headers, and inline descriptions. Always include a content.replace('\uff1a', ':') step in the final cleanup pass. Also check for fullwidth semicolons ๏ผ (U+FF1B).
โ ๏ธ Never use assert new_jp == 0 in batch scripts โ an assertion on file #4 kills the entire batch, losing progress on files #1-3 and requiring differential recovery for files #5-8. Use print+continue instead (see references/jp-en-translation-batch.md โ Principle 1).
โ ๏ธ Split fully-JP files into a separate batch from mixed JP/EN files โ they have fundamentally different failure modes (full rewrite vs per-string match). Each batch type gets isolated debugging (see references/jp-en-translation-batch.md โ Principle 2).
โ ๏ธ Quote-escaping trap: str.replace() values containing apostrophes ('s possessives) break single-quoted Python strings. Writing the script to a temp file (instead of terminal("python3 -c ...")) only bypasses bash backtick substitution โ Python single-quote conflicts remain. Two fixes:
a. Use \u2019 (Unicode right single quotation mark) for apostrophes: 'NVIDIA\u2019s inference' avoids the conflict entirely.
b. Use triple-quoted Python strings ("""...""") for str.replace() values so single quotes inside are parsed as literal characters.
c. Escape with \' inside single-quoted strings: 'NVIDIA\\'s inference' โ works but requires backslash-escaped quotes which hurt readability.
Commit both approaches as habits: write complex scripts to /tmp/ via write_file, AND use \u2019 or triple-quoted strings for any line containing apostrophes.
โ ๏ธ Multi-line replace trap: A str.replace() with \n\n between heading and body can silently fail due to subtle whitespace differences. Use shorter, line-by-line str.replace() calls instead of multi-line spans. Debug with repr() on the target content to find exact characters.
See pitfalls in references/jp-en-translation-batch.md for details.
Process simplest to most complex โ fewer JP chars first, index pages last.
Log via execute_code with header/chrono splitting (never f.write(new_entry + content)).
See references/jp-en-translation-batch.md for the complete workflow, scan script, and pitfalls.
When the user asks a question about the wiki's domain:
โ Read index.md to identify relevant pages.
โก For wikis with 100+ pages, also search_files across all .md files
for key terms โ the index alone may miss relevant content.
โข Read the relevant pages using read_file.
โฃ Synthesize an answer from the compiled knowledge. Cite the wiki pages
you drew from: "Based on [[page-a]] and [[page-b]]..."
โค File valuable answers back โ if the answer is a substantial comparison,
deep dive, or novel synthesis, create a page in queries/ or comparisons/.
Don't file trivial lookups โ only answers that would be painful to re-derive.
Karpathy's insight: "Good answers can be filed back into the wiki as new pages. A comparison you asked for, an analysis, a connection you discovered โ these are valuable and shouldn't disappear into chat history. This way your explorations compound in the knowledge base just like ingested sources do." This is the philosophical foundation for why queries belong in the wiki, not just in chat history.
Query page format โ see references/query-page-format.md for the full template. Key elements:
Creating the first query page: If wiki/queries/ doesn't exist yet, create it with mkdir -p. The index.md's Queries section will say > No queries filed yet. โ replace this with the first entry. Subsequent queries get inserted alphabetically.
โฅ Update log.md with the query and whether it was filed.
When the user asks to lint, health-check, or audit the wiki:
โ Orphan pages: Find pages with no inbound [[wikilinks]] from other pages.
# Use execute_code for this โ programmatic scan across all wiki pages
import os, re
from collections import defaultdict
wiki = "<WIKI_PATH>"
# Scan all .md files in entities/, concepts/, comparisons/, queries/
# Extract all [[wikilinks]] โ build inbound link map
# Pages with zero inbound links are orphans
โก Broken wikilinks: Find [[links]] that point to pages that don't exist.
โข Index completeness: Every wiki page should appear in index.md. Compare
the filesystem against index entries.
โฃ Frontmatter validation: Every wiki page must have all required fields (title, created, updated, type, tags, sources). Tags must be in the taxonomy.
โค Stale content: Pages whose updated date is >90 days older than the most
recent source that mentions the same entities.
โฅ Contradictions: Pages on the same topic with conflicting claims. Look for
pages that share tags/entities but state different facts. Surface all pages
with contested: true or contradictions: frontmatter for user review.
โฆ Quality signals: List pages with confidence: low and any page that cites
only a single source but has no confidence field set โ these are candidates
for either finding corroboration or demoting to confidence: medium.
โง Source drift: For each file in raw/ with a sha256: frontmatter, recompute
the hash and flag mismatches. Mismatches indicate the raw file was edited
(shouldn't happen โ raw/ is immutable) or ingested from a URL that has since
changed. Not a hard error, but worth reporting.
โจ Page size: Flag pages over 200 lines โ candidates for splitting.
โฉ Tag audit: List all tags in use, flag any not in the SCHEMA.md taxonomy.
โช Log rotation: If log.md exceeds 500 entries, rotate it.
โซ Report findings with specific file paths and suggested actions, grouped by severity (broken links > orphans > source drift > contested pages > stale content > style issues).
โฌ Append to log.md: ## [YYYY-MM-DD] lint | N issues found
When facing a large backlog of articles (500+) to potentially ingest:
blogwatcher-cli articles or equivalentsearch_files "new-page-name.md" path="$WIKI" target="files"
When the user asks to write original content (blog posts, essays, analyses, opinion pieces) sourced primarily from the wiki knowledge base:
โ Search the wiki thoroughly โ use search_files across concepts/, entities/,
comparisons/, and raw/articles/ for the topic and related terms. Cast a wide net
initially; narrow after you see what's available.
โก Read the key pages in depth โ prioritize concept pages with high cross-reference density and entity pages with documented philosophies. Don't skim โ the best synthesis comes from deeply understanding a few sources, not shallowly scanning many.
โข Supplement with web search โ only when wiki knowledge is thin or you need a specific external claim to bolster an argument. Prefer authoritative sources (arXiv papers, company blogs, recognized thought leaders). Cite these alongside wiki sources.
โฃ Compose the essay โ write in the user's language. Match the requested tone (lively/ๆดปใๆดปใ, academic, conversational, provocative). Structure as a narrative journey: thesis โ evidence โ counterpoint โ synthesis. Quote primary sources directly where they're vivid or iconic.
โค Cite everything โ every major claim needs a source. Wiki pages get [[wikilinks]],
external URLs get inline links. Distinguish between the author's words and your analysis.
For a wiki-sourced essay, the ratio should be ~70% wiki + ~30% external/web.
โฅ Save as a raw article โ use wiki/raw/articles/ with the pattern
YYYY-MM-DD_hermes_{topic-slug}.md for original content. Include YAML frontmatter
with title, date, author: Hermes (kzinmr's AI Topics), tags, and sources
(listing both wiki pages and external URLs).
โฆ Update log.md with the new article, key sources referenced, and a brief summary. Commit and push.
Quality guidelines:
When NOT to synthesize:
When running as a scheduled cron job to discover and fill knowledge gaps (no user-provided sources):
โ Research trending topics โ use web_search to find recent AI/ML announcements,
arXiv papers (peer-reviewed only), and official blog posts from major labs.
Target 3-5 topics the wiki likely hasn't covered.
โก Cross-reference against the wiki โ use execute_code to check which
topics already have pages. Run the full pattern from references/cross-reference-check.md
(candidate list โ os.walk() scan โ sorted FOUND/MISSING report with counts and paths).
Additional check: after the filesystem scan, grep index.md for near-matches of
each MISSING candidate's core slug (e.g., search nvidia-nemotron for candidate
nvidia-nemotron-3). The index may list a canonical full name like
nvidia-nemotron-3-nano-omni that your simplified candidate slug won't match on
os.path.exists(). This prevents creating duplicate pages with different slugs
for the same entity.
โข Extract original sources โ use web_extract on official sources (company blogs,
arXiv abstracts, tech reports). Pass all URLs in a single web_extract call to batch-fetch.
Skip aggregator sites and second-hand summaries. If web_extract returns truncated content
with "LLM summarization timed out", use browser_navigate on those URLs individually
as a fallback โ it produces full text snapshots including all paragraphs, lists, and quotes.
Note: paywalled sources (Fortune, beehiiv, some NYT) may return sparse content โ
cross-reference with alternative sources.
โฃ Save raw articles first โ write_file to wiki/raw/articles/ with descriptive
names (YYYY-MM-DD_source-topic.md).
โค Create wiki pages โ batch the creations. Each page must have: frontmatter,
4-6 cross-references, tags from the SCHEMA.md taxonomy.
โ ๏ธ Cross-reference pre-flight: After writing all pages, verify ALL [[wikilinks]]
in new pages resolve to actual files BEFORE committing. Run the verification script
from references/cross-reference-prefight.md and patch any broken links immediately.
This avoids the common "commit โ lint finds broken links โ patch cycle" that costs
3-4 extra tool calls per broken link.
โฅ Update navigation โ use patch to insert new index entries alphabetically
(see "Patch-based index insertion" below). Update section counts and total pages.
โฆ Validate tags before commit โ Before git commit, verify ALL tags in ALL new/updated pages exist in SCHEMA.md. The pre-commit hook will catch violations, but catching them proactively avoids a blocked commit and retry cycle. Use this checklist:
Predict the violations (active-crawl specific): The #1 cause of tag violations in active-crawl is new company/product names used as tags. When you create entity pages for newly discovered companies/products (Cohere, Stability AI, Antigravity, etc.), those names MUST be in SCHEMA.md's People/Orgs or Products categories. Before writing any page, scan your planned tags for company/product identifiers and check whether SCHEMA.md already lists them. If not, add them to the relevant category line FIRST, then write the page with those tags.
General checklist:
tags: line with a mental check against SCHEMA.md categoriesmusic, typescript) โ check SCHEMA.md; if missing, either add or replace with existing near-equivalent (audio-generation, developer-tooling)patch calls on the relevant category linesgit add wiki/ and commit. If the hook still blocks, check the error output for ALL offending files (not just yours โ pre-existing staged violations from other pipelines also block)โง Log and commit โ single log entry covering the batch, commit + push.
For large index.md files (500+ lines), use patch with old_string/new_string
to insert new entries alphabetically rather than reading the entire file.
When making 10+ index changes (insertions + description updates), a single execute_code call with str.replace() across all updates can be more efficient than issuing 10+ individual patch calls:
with open(index_path) as f:
content = f.read()
# Read exact full lines BEFORE writing code
old = '- [[entities/daytona-io]] โ old summary...'
new = '- [[entities/daytona-io]] โ new summary with podcast details...'
content = content.replace(old, new)
# For insertions, use a 2-line anchor (line before + line after)
old_anchor = '- [[entities/evanhahn-com]] โ **Blog** | evanhahn.com |\n- [[entities/fabiensanglard-net]] โ **Blog** | fabiensanglard.net |'
new_with_insert = '- [[entities/evanhahn-com]] โ **Blog** | evanhahn.com |\n- [[entities/exa]] โ New entry summary...\n- [[entities/fabiensanglard-net]] โ **Blog** | fabiensanglard.net |'
content = content.replace(old_anchor, new_with_insert)
with open(index_path, 'w') as f:
f.write(content)
CRITICAL - Anchor string accuracy: Every old_string anchor MUST be the exact full line as it appears in the file โ no truncation, no abbreviation. A common failure mode is using a shortened anchor (e.g., stopping at the first sentence of a multi-sentence line), which silently fails to match. Read the exact lines with read_file(offset=N, limit=5) before writing the str.replace() code. If an anchor doesn't match:
if old in content: print("OK") else: print("FAIL") for each replacementWhen to use which approach:
patch calls (atomic, explicit, easy to verify)execute_code with str.replace() (fewer tool calls, batch-verifiable)CRITICAL: Do NOT compute insertion points programmatically. The index at scale
is NOT strictly alphabetical โ entries drift over time as different agents and
pipelines add content. A programmatic "find first alphabetical successor" algorithm
will almost always place entries in the wrong position, creating duplicates that
require cleanup. Instead, use read_file with the target offset to visually
confirm the 2-3 surrounding lines, then use those exact lines as the old_string
anchor.
Discovery step (do this first): Read the relevant section of index.md with
read_file(offset=N, limit=20) to find the exact anchor lines. Do NOT use
execute_code with os.walk() or regex-based successor computation โ it will
place entries incorrectly in non-alphabetical sections. See
references/index-insertion-discovery.md for the full pattern and failure cases.
# Instead of read_file + edit + write_file:
patch(
old_string="- [[entities/gemini]] โ summary...\n- [[entities/gemma-4]] โ summary...",
new_string="- [[entities/gemini]] โ summary...\n- [[entities/gemini-enterprise-agent-platform]] โ ...\n- [[entities/gemma-4]] โ summary...",
path="~/wiki/index.md"
)
Use 2-3 surrounding lines as the old_string anchor to ensure uniqueness.
Also update the header counts and section counts with separate patch calls.
Advantages over read+write:
When using delegate_task for wiki operations:
Use the canonical profile paths:
~/ai-topics~/wiki~/.hermes/scripts~/wiki is the source of truth and is expected to resolve to the git-managed
wiki under ~/ai-topics/wiki. Do not infer or write to alternate paths such as
/opt/data/.hermes/home/wiki, /opt/data/home/wiki, or historical
~/.hermes/hermes-agent/wiki locations.
If a delegated context has an isolated HOME, it must still target the canonical
paths above. Treat any ~/.hermes/home/... path that appears during execution as
a runtime artifact, not a second wiki.
for f in os.listdir(src_dir):
if f.endswith('.md'):
src_path = os.path.join(src_dir, f)
dst_path = os.path.join(dst_dir, f)
if not os.path.exists(dst_path):
shutil.copy2(src_path, dst_path)
print(f"COPIED: {f}")
elif os.path.getsize(src_path) > os.path.getsize(dst_path):
shutil.copy2(src_path, dst_path)
print(f"UPDATED: {f}")
**Prevention strategy:** When writing subagent instructions, explicitly state the full absolute path:
- Use `/opt/data/ai-topics/wiki/concepts/` (NOT `~/wiki/concepts/`)
- Or include the copy script in the subagent's instructions as a final step
**Verification is still required** โ even with absolute paths, subagents may ignore them.
## Git Merge Corruption Recovery
When a merge or automated edit corrupts wiki files (especially `log.md`), follow this recovery procedure:
### Symptoms to look for
1. **Line number prefixes in content:** Lines like ` 1|## 2026-04-15` indicate `read_file` output was accidentally pasted into the file
2. **Missing sections:** A recent commit's entry disappeared from `log.md` after a merge
3. **Orphaned headers:** `### Updated Entity Pages` without its parent `## YYYY-MM-DD โ Title`
4. **Missing `---` separators:** Sections run together without dividers
### Recovery procedure
โ **Check for git conflict markers first:**
```bash
grep -n "<<<<<<\|======\|>>>>>>" wiki/log.md
If found, this is a standard merge conflict โ resolve with standard git workflow.
โก If no conflict markers but content is missing, find the lost content in git history:
# Find the commit that originally added the missing section
git log --all --oneline -- wiki/log.md | grep -i "<keyword>"
# Recover the content from that commit
git show <commit-hash>:wiki/log.md
โข Fix line number corruption: Remove lines matching the pattern ^\s*\d+| that were accidentally embedded:
import re
with open("wiki/log.md") as f:
content = f.read()
# Remove corrupted line-number-prefixed lines
lines = content.split('\n')
cleaned = [line for line in lines if not re.match(r'^\s*\d+|##', line)]
โฃ Restore missing section headers: If a section lost its ## YYYY-MM-DD โ Title header (common when line-number corruption removes it), add it back before the orphaned ### subsections.
โค Add missing --- separators: Ensure sections are properly delimited.
โฅ Verify the fix:
# Check no more corruption patterns
grep -c "^\s*\d+|" wiki/log.md
# Verify section count makes sense
grep -c "^## 2026" wiki/log.md
โฆ Commit and push:
cd ~/ai-topics && git add wiki/log.md && git commit -m "fix: restore <section> and remove line number corruption" && git push
read_file output directly into wiki files โ the line number prefixes ( 1|) corrupt the markdownread_file().content programmatically for file modification โ the returned content field already contains line-number prefixes. result = read_file(path); lines = result["content"].split("\\n") and writing those back embeds prefixes in the file. Instead, always use direct Python file I/O: with open(path) as f: content = f.read() or lines = f.readlines(). This is the same reason patch is preferred over read_file+write_file for edits โ see "Patch-Based Index Insertion" for the correct approach.log.md structure โ merges can silently drop sectionsgit show <commit>:path to recover lost content rather than reconstructing from memory# Find pages by content
search_files "transformer" path="$WIKI" file_glob="*.md"
# Find pages by filename
search_files "*.md" target="files" path="$WIKI"
# Find pages by tag
search_files "tags:.*alignment" path="$WIKI" file_glob="*.md"
# Recent activity
read_file "$WIKI/log.md" offset=<last 20 lines>
When ingesting multiple sources at once, batch the updates:
Batch page creation via execute_code: When creating 3+ wiki pages in a session,
consolidate the file writes into a single execute_code invocation that writes all pages
(raw articles, entity pages, concept pages) in one Python script. This is more token-efficient
than N separate write_file calls. Use Python's os.makedirs(exist_ok=True) for directory
creation and open(path, 'w') for each file. After the batch, verify all files exist with
os.path.exists() checks in the same script. Similarly, batch index insertions by issuing
multiple patch calls in parallel (they target different sections and don't conflict).
Batch enrichment via execute_code (preferred over N ร write_file): When enriching
N existing entity/concept pages with new sections (from raw articles), use a SINGLE
execute_code call with Python's with open(path) as f: content = f.read() for reading
each file, apply modifications (string replacements, section insertions, updated date bumps),
then with open(path, 'w') as f: f.write(content). This pattern:
write_file calls (token overhead)print() summary at the endupdated date, consistent source references)if 'target section header' not in content: guards to prevent idempotency errors.โ ๏ธ Split strategy when monolithic execute_code fails: When the enrichment script grows large (10+ files, nested strings with quotes), Python syntax errors from complex .replace() calls with escaped quotes are common. Also fails on large dict literals with multi-line f-string values โ the parser loses track of closing braces, producing SyntaxError: '{' was never closed. This is a distinct failure mode from quote escaping; the fix is the same split strategy below. Recovery pattern:
write_file calls โ these are simple, independent, and never trigger quote-escape issues.execute_code call that reads + modifies + writes each file using Python's with open() pattern.write_file bypasses the Python-internal quoting problem entirely for new content, while execute_code modifications avoid it by never placing the full content string inside Python string literals โ they read from disk instead.โ ๏ธ Markdown table pipe prefix mismatch (silent str.replace failure): Before using str.replace() to modify a markdown table row (e.g., adding a feature to a feature table), check whether the file uses single | or double || pipe prefixes for table rows. Files created by different pipelines may differ: some use | Feature | Detail | (single pipe), others || Feature | Detail || (double pipe). A str.replace() targeting || Claude Opus 4.8 | ... || will silently do nothing on a file using single-pipe format. Detection after failure: read_file(offset=N, limit=3) shows the actual prefix. Fix: When the first str.replace fails, use repr() on the target lines to discover the exact pipe prefix. Prevention: Always read the exact table row via read_file before writing the replacement strings โ match the actual pipe count, don't assume ||. This was discovered in a 9-file batch enrichment where 3 replacements silently failed due to single-pipe vs double-pipe mismatch.
โ ๏ธ Frontmatter updated field may be entirely absent: Some older entity pages (especially those created by the blog-wiki pipeline or early agent sessions) have a created field but NO updated field. A str.replace() targeting "updated: 2026-05-11" will silently fail โ there's nothing to replace. Detection: Before running the batch, check whether "updated:" exists in each target file's frontmatter (first 15 lines). Fix: If updated: is absent, add it after created: using content.replace(f"created: {created_date}\n", f"created: {created_date}\nupdated: {today}\n"). If the roles: or education: keys appear between created and where updated should go, the frontmatter has non-standard ordering โ handle by inserting after created: regardless. Prevention: Include a pre-flight check in every batch enrichment: read the first 15 lines of each target file and assert the presence of "updated:" โ or add it if missing.
โ ๏ธ Section insertion anchor mismatch โ heading vs list item: When inserting new content before an existing section, verify the anchor text is actually a heading (### Section Title) and not a list item (- **Section Title**:). These look identical at a glance but have different match behavior. Common in entity pages where feature descriptions use - **Feature Name**: Description format rather than ### Feature Name. A str.replace() targeting ### Mid-conversation system messages: will silently fail against the actual - **Mid-conversation system messages**:. Detection after failure: read_file(offset=N, limit=3) reveals the actual line prefix. Prevention: Read the exact 3 lines around the insertion point before writing the replacement string; copy the exact prefix (including dash, stars, indent level) for the anchor.
When content is fully superseded or the domain scope changes:
_archive/ directory if it doesn't exist_archive/ with its original path (e.g., _archive/entities/old-page.md)index.mdThe wiki directory works as an Obsidian vault out of the box:
[[wikilinks]] render as clickable linksraw/assets/ folder holds images referenced via ![[image.png]]For best results:
raw/assets/TABLE tags FROM "entities" WHERE contains(tags, "company")If using the Obsidian skill alongside this one, set OBSIDIAN_VAULT_PATH to the
same directory as the wiki path.
On machines without a display, use obsidian-headless instead of the desktop app.
It syncs vaults via Obsidian Sync without a GUI โ perfect for agents running on
servers that write to the wiki while Obsidian desktop reads it on another device.
Setup:
# Requires Node.js 22+
npm install -g obsidian-headless
# Login (requires Obsidian account with Sync subscription)
ob login --email <email> --password '<password>'
# Create a remote vault for the wiki
ob sync-create-remote --name "LLM Wiki"
# Connect the wiki directory to the vault
cd ~/wiki
ob sync-setup --vault "<vault-id>"
# Initial sync
ob sync
# Continuous sync (foreground โ use systemd for background)
ob sync --continuous
Continuous background sync via systemd:
# ~/.config/systemd/user/obsidian-wiki-sync.service
[Unit]
Description=Obsidian LLM Wiki Sync
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/path/to/ob sync --continuous
WorkingDirectory=/home/user/wiki
Restart=on-failure
RestartSec=10
[Install]
WantedBy=default.target
systemctl --user daemon-reload
systemctl --user enable --now obsidian-wiki-sync
# Enable linger so sync survives logout:
sudo loginctl enable-linger $USER
This lets the agent write to ~/wiki on a server while you browse the same
vault in Obsidian on your laptop/phone โ changes appear within seconds.
TAG GATE (HARD RULE โ TRIGGERS ON EVERY WIKI WRITE) โ When creating or updating ANY wiki page (entity, concept, comparison, query), every tag in the frontmatter MUST come from wiki/SCHEMA.md's Tag Taxonomy section. Before writing a page: (1) read SCHEMA.md's taxonomy, (2) ensure every tag you plan to use exists there. If you need a genuinely new tag category, add it to SCHEMA.md FIRST, then use it. NEVER use ad-hoc tags. Composite kebab-case tags (5+ hyphen-joined words like cognition-devin-memory-tool-claude-code) are ALWAYS errors โ decompose them into individual canonical tags. A pre-commit hook blocks commits with non-SCHEMA tags; if you see a TAG TAXONOMY VIOLATION block, fix the tags before committing.
| pipe prefix corruption in log.md from patch โ When using patch to append entries to log.md, if the old_string or new_string contains a trailing | character (from markdown table formatting or clipboard artifacts), that | gets written into the file as a standalone line. This produces entries like:
- Total pages: 1834
|
## [2026-05-13] dreaming | ...
Detection: grep -n '^|$' wiki/log.md finds standalone pipe lines. Fix: sed -i '/^|$/d' wiki/log.md or Python content = re.sub(r'\n\|\n', '\n', content). After fixing, verify with grep -c '^|$' wiki/log.md returns 0. To prevent, never include bare | at the end of new_string in patch calls โ the | suffix from line-number-prefixed read_file output is the most common source.
In-page content duplication (repeated sections in same file) โ Entity pages can develop repeated sections when multiple ingest pipelines append the same topic (e.g., "The Mismeasure of Open Source" appearing 3 times in entities/andrew-nesbitt.md). This happens when different pipelines enrich the same page without checking whether the section already exists. Detection: grep -c '^###' on a page to flag unusually high section counts for its subject breadth. For specific topics, search for repeated header strings. Fix: Use Python to find duplicate section boundaries:
import re
with open(path) as f: content = f.read()
# Find duplicate sections by header text
headers = re.findall(r'^### .+', content, re.MULTILINE)
seen = {}
for h in headers:
if h in seen:
print(f"DUPLICATE: {h}")
seen[h] = True
# Remove all but first occurrence of each duplicate section
# Find the start of each duplicate (after first) and remove to next header
Prevention pipeline: When enriching an existing page with blog-post-specific content, always search the page for the article's title or key phrases first (grep -ci "exact article title") before adding a new section. This applies to both automated cron pipelines and manual enrichments.
SOURCES GATE (HARD RULE โ TRIGGERS ON EVERY WIKI WRITE) โ Every wiki page MUST have a sources: field in its frontmatter, even if empty (sources: []). When creating a page from a raw article, set sources: [raw/articles/<filename>]. When creating from web extraction or synthesis, list the URLs: sources: [https://example.com/article]. Never omit the sources field โ this was the single largest frontmatter gap discovered in 2026-05-13 audit (770+ of 810 broken pages were missing sources). If unsure, use sources: [] rather than omitting it entirely.
Never modify files in raw/ โ sources are immutable. Corrections go in wiki pages.
Always orient first โ read SCHEMA + index + recent log before any operation in a new session. Skipping this causes duplicates and missed cross-references.
Always update index.md and log.md โ skipping this makes the wiki degrade. These are the navigational backbone.
Don't create pages for passing mentions โ follow the Page Thresholds in SCHEMA.md. A name appearing once in a footnote doesn't warrant an entity page.
Don't create pages without cross-references โ isolated pages are invisible. Every page must link to at least 2 other pages.
Frontmatter is required โ it enables search, filtering, and staleness detection.
Tags must come from the taxonomy โ freeform tags decay into noise. Add new tags to SCHEMA.md first, then use them.
Keep pages scannable โ a wiki page should be readable in 30 seconds. Split pages over 200 lines. Move detailed analysis to dedicated deep-dive pages.
Ask before mass-updating โ if an ingest would touch 10+ existing pages, confirm the scope with the user first.
Rotate the log โ when log.md exceeds 500 entries, rename it log-YYYY.md and start fresh.
The agent should check log size during lint.
Handle contradictions explicitly โ don't silently overwrite. Note both claims with dates, mark in frontmatter, flag for user review.
execute_code file writes are sandboxed โ use write_file for wiki pages: execute_code with
Python's open(path, 'w') creates files in a temporary sandbox, NOT on the real filesystem.
The script will report "CREATED: entities/foo.md" and os.path.exists() returns True, but the
file won't exist when you try git add or ls from terminal. For creating or modifying wiki
pages, always use write_file with absolute canonical paths (/opt/data/ai-topics/wiki/...).
execute_code is still fine for reading, processing, analysis, and cross-referencing โ just not
for writing wiki files. This also applies to raw article files under wiki/raw/articles/.
Recovery: after discovering the files are missing, use write_file with the same content
to recreate them on the real filesystem.
Verify before claiming completion โ always check that files actually exist after delegate_task
or any async operation before telling the user work is done. Use search_files to confirm.
Don't over-promise on article counts โ when analyzing a large backlog (2,000+ articles), report the filtered count and processing plan first, don't claim pages are created until they're verified.
Subagent iteration budgets are real โ max_iterations=20 allows ~5-7 page creations. For larger batches, split across multiple subagents or increase max_iterations.
search_files is unreliable for wiki directory discovery โ it may return 0 results for files that
definitely exist (e.g., entity pages like simon-willison.md). Use Python os.walk() or
os.path.exists() via execute_code instead when locating specific wiki pages.
Entity file-to-identity check โ slugs are ambiguous: os.path.exists("pi.md") returning true does NOT mean Pinecone has an entity page โ it could be the Pi coding agent. Similarly, contextarena.md is a benchmark, not Arena the company. When checking whether an entity already has a page, always read the page's title/overview (not just check file existence) to confirm the slug maps to the expected entity. File-name collisions between unrelated entities sharing similar names are common in large wikis.
Slug near-matches during cross-reference: when checking for duplicates via os.path.exists(), a candidate slug like nvidia-nemotron-3 won't match the canonical nvidia-nemotron-3-nano-omni. The cross-reference script must also scan index.md for near-match entries โ grep for the candidate's core slug (e.g., nvidia-nemotron) across the index before creating a new page. The index often has the canonical full name while your simplified candidate slug misses it. Use search_files on the index with a substring of the candidate name as a secondary check.
|- pipe table syntax corruption โ automated tools or scripts can render index.md entries as markdown table rows (|- [[entities/foo]] โ summary). This corrupts ~200+ lines at scale. Detection: grep -c '^|- \[\[' wiki/index.md.
Fix with Python (one-shot, all lines):
import re
with open("wiki/index.md") as f: content = f.read()
fixed = re.sub(r'^\|-\\s+\[\[(?:entities|concepts|comparisons|queries)/',
lambda m: '- [[' + m.group(0)[4:], content, flags=re.MULTILINE)
with open("wiki/index.md", 'w') as f: f.write(fixed)
Post-fix verification: commit must say โ wiki/index.md clean (N lines) and NOT flag any pipe-prefixed list item warnings.
Index header count math is unreliable โ "Total pages: N" frequently diverges from actual filesystem count (discrepancies of 600+ observed at 1,754 pages; Concepts section showed 459 vs actual 1,064). "Full entries" + "Stubs" may not sum to "Total". Always verify with os.walk() or ls ~/wiki/concepts/*.md | wc -l on the filesystem, never trust the header alone. Severity at scale: At 5,000+ pages, discrepancies compound further.
Extension-less wikilinks to _index.md โ links like [[concepts/_index]] or [[agentic-engineering/_index]] will NOT resolve on case-sensitive filesystems. The target file is _index.md, not _index. Fix: append .md or use [[agentic-engineering/_index|_index.md]] syntax.
Log prepending via patch is dangerous โ two failure modes:
โ Header swallowing (this session): When prepending a new log entry before an existing ## [YYYY-MM-DD] entry, using that entry's header line as the old_string anchor will silently delete the anchor header. The patch replaces the matched text with new_string, so the existing entry's header disappears and its sub-sections (### Pages Created, ### Pages Updated) become orphaned under your new entry. Example of what happens: the file goes from ## [2026-05-14] enrich | new entry\n\n### Pages Updated\n... to having the original ## [2026-05-14] ingest | existing entry line gone entirely, with ### Pages Created and ### Pages Updated now misattributed to your enrich entry.
โก Duplication: When new_string accidentally includes a second copy of the anchor line, both copies are written (already documented).
Correct approach for log prepending: Do NOT use patch โ use execute_code with Python to insert lines at a known position. Pattern:
with open(path) as f: lines = f.readlines()
# Insert new entry after the header block
# โ ๏ธ CRITICAL: Find where the header ends, do NOT hardcode line 7.
# The # Wiki Log header + metadata lines are followed by a blank line,
# then the first ## [YYYY-MM-DD] chronological entry.
# Find the first ## [ entry to split header from chrono content:
chrono_start = None
for i, line in enumerate(lines):
# Handle both ## [YYYY-MM-DD] and ## YYYY-MM-DD formats (mixed format)
import re
if line.startswith('## ['):
chrono_start = i
break
if line.startswith('## ') and re.match(r'## \d{4}-\d{2}-\d{2}', line):
chrono_start = i
break
# If still None, fall back to first ## line (edge case: no date-entries yet)
if chrono_start is None:
for i, line in enumerate(lines):
if line.startswith('## '):
chrono_start = i
break
# โ ๏ธ CRITICAL: Do NOT filter chrono_start by date. The first `## [` in the file is
# the correct split point, even if it shares today's date. Adding `and not
# line.startswith('## [2026-05-24]')` would skip same-day entries and place YOUR
# new entry BELOW them instead of at the top of the chronological section, breaking
# newest-first ordering. The chrono_start loop must be a simple first-match search
# with no date-based exclusions.
# header_block = lines[:chrono_start], chrono_entries = lines[chrono_start:]
# โ ๏ธ AFTER writing, ALWAYS verify the header survived:
# with open(path) as f: first_line = f.readline()
# assert first_line.strip() == '# Wiki Log', f"Header buried! Got: {first_line.strip()}"
header_block = lines[:chrono_start]
new_entry = ["## [YYYY-MM-DD] enrich | subject\n", "\n", "### Pages Updated\n", ...]
chrono_entries = lines[chrono_start:]
lines = header_block + new_entry + ["\n", "---\n", "\n"] + chrono_entries
with open(path, 'w') as f: f.writelines(lines)
โ ๏ธ WRITELINES TRAP: f.readlines() vs content.split('\n') โ The code above uses f.readlines() which preserves trailing \n on each line. If you instead use content.split('\n') to read the file, trailing newlines are stripped. Passing that stripped list to f.writelines() produces a corrupted file where lines merge together (e.g. # Wiki Log> Chronological record...Append-only.## [YYYY-MM-DD]). Use f.readlines() for readability in log prepending OR use '\n'.join() + a single f.write() call when working from split('\n') output. The post-write assertion (assert f.readline().strip() == '# Wiki Log') catches this corruption immediately.
Recovery from writelines/split-newline corruption: When the header merges into a single line, try one of these fix patterns:
# Light fix for simple merged-header cases
content = content.replace('# Wiki Log>', '# Wiki Log\n>')
content = content.replace('Append-only.##', 'Append-only.\n\n##')
# Full reconstruction from git (for complex corruption)
import subprocess
result = subprocess.run(["git", "show", "HEAD:wiki/log.md"],
capture_output=True, text=True, timeout=10)
After fixing, verify with head -4 wiki/log.md that # Wiki Log, the metadata line, a blank line, and ## [YYYY-MM-DD] are on separate lines.
โ ๏ธ FAILURE MODE: DO NOT use f.write(new_entry + content) โ this simple prepend pattern buries the # Wiki Log header because content starts with the header, and prepending new_entry before it pushes the header to line 3+. The resulting file starts with ## [YYYY-MM-DD] instead of # Wiki Log, violating the canonical structure. This happened in the 2026-05-20 health-fix session: the header ended up at line 37 and required a multi-step recovery (rescue header from middle of file, reconstruct with correct ordering). Always insert AFTER the header block using header/chrono splitting as shown above.
โ ๏ธ CRITICAL sub-pitfall โ do NOT use content.index() for position finding. When using re.search() to find a match position, do NOT call content.index(match.group(2), match.start()) to compute the replacement range. content.index() returns the FIRST occurrence of the substring from the given start position โ but if match.group(2) is a common pattern like ## [YYYY-MM-DD], it will match the very first entry in the file (not the one adjacent to your search match), causing wholesale truncation of everything below the first entry. This was demonstrated in the 2026-05-20 watchdog session: attempting to remove an orphan ### line using content.index('## [', match.start()) truncated 1,100 lines, requiring git checkout recovery. Always extract the exact position from the match object itself: use match.end(2) - len(match.group(2)) or store the position in a variable from the regex match directly.
โ ๏ธ POST-WRITE VERIFICATION: After writing log.md, ALWAYS verify:
with open(path) as f: first_line = f.readline()
assert first_line.strip() == '# Wiki Log', f"Header buried! First line: {first_line.strip()}"
# Also verify: grep -c '^# Wiki Log' = 1, first ## [ entry follows header
โ ๏ธ SUB-PITFALL: Backtick-wrapped ## [ in the Format line is a chrono_start false positive. The > Format: \## [YYYY-MM-DD] action | subject`metadata line contains the literal string## [inside backtick code. When findingchrono_startviacontent.find("## [")orcontent.index("## [", start), this metadata match fires BEFORE any real chronological entry, causing the split to happen inside the backtick span. The result: the Format line is truncated to > Format: `(everything after the first backtick is lost). **Detection**:head -4 wiki/log.mdshows> Format: `instead of> Format: `## [YYYY-MM-DD] action | subject`. **Fix**: Use a regex that excludes backtick-wrapped matches, e.g., re.search(r'(?<!`)## [', content)or split on the first## [that appears on its own line (after a\n, not after backtick on the Format line). **Prevention**: In the chrono_startdetection loop, skip lines where## [appears inside a backtick code span by checking the line starts with> ` (blockquote prefix used for metadata). Pattern:
if line.startswith('## [') and not line.startswith('> '): # Skip > Format: `## [...
chrono_start = i
break
After any log.md write, always verify with head -4 ~/wiki/log.md that the Format line is complete, not truncated.
If you must use patch instead of execute_code, use a multi-line anchor spanning the gap between entries (e.g., the --- separator + next entry's full header + first sub-section line) to ensure uniqueness and prevent swallowing. After any log edit, verify the first ~30 lines with read_file(limit=30) to confirm entries are properly separated.
Verify log.md structure after rotation โ log rotation scripts can produce a duplicate
header section: the first # Wiki Log header + rotate entry + --- separator, followed
by a second # Wiki Log header + duplicate rotate entry. Detection: grep -c "^# Wiki Log" wiki/log.md
must return exactly 1. Fix: remove everything from the second # Wiki Log occurrence to
the beginning of the next legitimate ## [YYYY-MM-DD] entry. See wiki-graph-health skill's
references/log-rotation.md for the full procedure and Python fix script.
Missing # Wiki Log header (silent loss) โ The entire # Wiki Log header block (title + metadata lines) can go missing from log.md due to prior execute_code log-prepending operations that use wrong line offsets. The file then starts directly with ## [YYYY-MM-DD] entries. Detection: head -1 wiki/log.md should return # Wiki Log, not a date-stamped header. Recovery: use a large patch that inserts the header block + your new entry simultaneously before the first existing entry. Pattern: old_string = the first ## [YYYY-MM-DD] entry's header line + first subsection, new_string = full # Wiki Log header block + your new entry + the existing first entry. This fixes both the missing header and inserts your entry in a single atomic operation. Prevention: never assume line 7 is the insert point for execute_code log prepending โ read the first 5 lines first to confirm the header is present.
Watch for index entry drops when patching โ when inserting into index.md via patch,
never guess what's between two anchor lines. The old_string must match the file EXACTLY.
If you assume recursive-self-improvement immediately follows the section header but the
file actually has accelerate in between, patch will silently drop accelerate and
possibly create duplicates. Use read_file with the exact offset to confirm the 2-3 lines
forming your old_string anchor before calling patch. After every index patch, verify
the section with read_file before proceeding. See references/index-insertion-discovery.md
for the full pattern and failure case.
Triple bracket corruption ([[[): A variant discovered 2026-05-10 where index entries gain a third opening bracket: [[[concepts/foo]] instead of [[concepts/foo]]. This renders wikilinks unparseable by Obsidian/wiki tools. Detection: grep -c '[[\[' wiki/index.md. Fix: content.replace('[[[', '[[') โ always safe since triple brackets are never intentional.
Index entry points to wrong directory: Entry says [[concepts/slug]] but file lives in entities/ (or vice versa). Causes entry to appear "missing" from filesystem. Detection: For each [[dir/slug]], check os.path.exists(wiki/dir/slug.md). If false, check the other namespace. Fix: Replace wrong namespace prefix with correct one.
How this happens during active-crawl creation: The agent batch-creates pages across namespaces (e.g., concepts/gemini-spark.md and entities/qwen-3-7-max.md), then inserts each index entry next to alphabetically-adjacent entries. If gemini-spark is alphabetically closest to entries in the Entities section (e.g., gemini-3-5-flash, gemma-4), the agent naturally inserts it there โ but writes [[entities/gemini-spark]] instead of [[concepts/gemini-spark]]. Prevention: After batch-creating pages, verify EACH index entry's namespace prefix matches the file's actual directory. In the execute_code batch, cross-check: for entry in new_entries: assert entry.startswith(f"[[{actual_dir}/").
"Found 2 matches" patch failures due to list-marker inconsistency โ when patch returns
"Found 2 matches for old_string" but search_files shows each line appearing only once,
the culprit is often a mix of - (dash space) and |- (pipe dash space) list markers in
index.md. The patch tool sees both patterns as valid anchor candidates even though they
differ. Fix: (1) read the target section with read_file(offset=N, limit=6) to see the
exact content characters, (2) use a 4-line old_string (including one line before and after
the target pair) to make the match unique, or (3) run the pipe-table fix recipe first
(see |- pipe table syntax corruption pitfall) to normalize all list markers, then retry
the patch.
Newsletter triage JSON parsing failure โ when newsletter-wiki-ingest receives
failed to parse JSON response from newsletter-triage output, the triage output is
embedded in a markdown-wrapped cron output file. Fallback: read
${HERMES_HOME}/cron/data/newsletter/triage_latest.json directly โ this is pure JSON
saved by the triage job. If that's missing, grep the .md output file for the JSON
block after the ## Response heading.
Same pattern for all pipelines โ the blog-wiki-ingest and dreaming-wiki-ingest
jobs face the identical issue. For blog: ${HERMES_HOME}/cron/data/blog_ingest/triage_latest.json.
For dreaming: ${HERMES_HOME}/cron/data/dreaming/triage_latest.json. The upstream triage
job saves JSON to the checkpoint path before the wiki-ingest runs; always read from the
checkpoint path, not from the cron output file.
web_extract blocked on public URLs (private network error) โ some hosting platforms (Quartz, certain CDNs) trigger this. Use the decision tree in references/content-extraction-fallbacks.md: search for alternative mirrors โ curl raw HTML โ try .md endpoint โ browser_navigate last resort.
web_extract returns only navigation chrome (silent failure) โ web_extract reports success but returns only menus/footers/cards with zero article body. Common on JS-rendered official blogs (Google DeepMind, OpenAI, Anthropic). On headless servers without Chrome, use web_search triangulation: query 2-3 angles to find third-party tech news coverage (MarkTechPost, The Register, The Decoder) that reproduces key quotes and data. See references/content-extraction-fallbacks.md step 4 for the full pattern and example.
Paywalled content is still usable โ beehiiv/substack paywalled articles with free previews often contain enough technical claims (model names, chip specs, utilization numbers) to justify wiki updates. Cross-reference with non-paywalled sources. Mark uncertain claims with qualifiers ("reports suggest", "rumored").
web_extract LLM summarization truncation โ long articles (especially news articles and
Substacks) frequently trigger: LLM summarization timed out. To fix: increase auxiliary.web_extract.timeout in config.yaml, or use a faster auxiliary model. Use browser_navigate for the full page. When this happens, the first ~5,000 characters are still
usable for extracting key facts. For richer content, use browser_navigate on the URL
(produces a full text snapshot including all paragraphs, lists, and quotes) โ but note that
browser_navigate requires Chrome/Chromium installed (agent-browser install). On headless
servers where Chrome isn't available, fall back to targeted web_search queries for the
specific model/product name to supplement the truncated content with specifications,
benchmarks, and architecture details from alternative sources. Treat web_extract as a fast
initial pass and browser_navigate/web_search as the fallback for pages that time out.
For large arXiv papers (100K+ chars) where web_extract times out on the HTML tab: download the arXiv HTML version
with curl, extract section headings to understand structure, then use Python regex keyword search to find
new/changed content. Full procedure in references/large-paper-extraction.md.
updated date bump frequently forgotten during execute_code enrichment โ When enriching N existing entity/concept pages via execute_code Python string replacements (not via write_file or patch), the updated date in YAML frontmatter is easy to skip because str.replace() or ad-hoc section insertion doesn't naturally touch the frontmatter block. Verification step updated=False in the post-commit checks is the tell: if the entity page's updated field wasn't bumped, the enrichment isn't recorded in the wiki's recency tracking. Fix: include an explicit date bump step in the same execute_code block โ read the frontmatter, find the updated: line, and replace it with today's date. Also add the bumped line to the file's sources field for proper source tracking. After writing, verify has_updated = "updated: YYYY-MM-DD" in content in the same script. This applies to ALL entity/concept enrichments, not just execute_code โ verify the updated field for every page touched before committing.
Tag taxonomy drift is inevitable at scale truncated content.
str.replace() failures due to file write-back โ execute_code calls are independent processes. Each execute_code call starts a fresh Python process. If you modify content in one call (e.g., via str.replace()), the next call reads the original file. Always write to disk in the SAME execute_code call:
with open(path) as f: content = f.read()
content = content.replace(old, new)
with open(path, 'w') as f: f.write(content)
After writing, verify in the same call: print(f"Wrote {len(content)} chars").
For complex Python scripts with nested quotes, write to a temp file. When the script has multiple str.replace() calls, f-strings with ''' or """, or backtick-wrapped literals (URLs, raw article paths), inline execute_code or terminal("python3 -c ...") can fail with SyntaxError or bash backtick substitution. Write the script to /tmp/ and run via terminal():
write_file(path="/tmp/fix_index.py", content=python_script)
terminal("python3 /tmp/fix_index.py")
Proven on 20-entry index batch (2026-05-27).
kebab-case tags (e.g., cognition-devin-memory-tool-claude-code-competitive-analysis) as
errors โ these should be decomposed into separate tags. Add frequently-used legitimate tags
to the taxonomy during lint, don't let them proliferate silently.
Tag extraction from SCHEMA.md requires multi-pattern matching โ the taxonomy section uses
mixed formats: backtick-wrapped inline lists (model, multimodal, text-generation), bullet
points with comma-separated tags, and multi-line continuations. A simple regex like
`([a-z-]+)` captures only ~60% of canonical tags. For accurate audits, parse each
category header line, extract all backtick-wrapped AND comma-separated tokens, and handle
line continuations. If your audit shows 150+ "violations" but SCHEMA.md clearly contains
those tags, your extraction logic is too narrow.
Case-sensitivity wikilink fix procedure โ when [[concepts/mooncake]] points to a file
named Mooncake.md on a case-sensitive filesystem:
mv wiki/concepts/Mooncake.md wiki/concepts/mooncase.mdgit add wiki/concepts/Mooncake.md wiki/concepts/mooncake.md
(Git needs to see both sides of the rename to track it properly)git commit -m "fix: rename Mooncake.md -> mooncase.md"search_files '\[\[concepts/Mooncake\]\]' โ patch to [[concepts/mooncake]]Index header count decays over time โ index.md header's "Total pages: N" quickly
becomes stale as pages are created. During lint, always verify:
len(os.listdir(entities/) + os.listdir(concepts/) + ...) == reported_count.
Discrepancies of 100+ pages are common at scale and should be auto-corrected.
Case sensitivity in wikilinks โ [[concepts/mooncake]] will NOT resolve to
concepts/Mooncake.md on case-sensitive filesystems (Linux servers). During broken link
checks, try case-insensitive basename matching as a secondary resolution strategy.
_index.md files are special โ subdirectory _index.md files (e.g., concepts/_index.md)
serve as synthesis hubs and may legitimately lack full frontmatter. During lint, exclude
files named _index.md from frontmatter-required checks, or give them a minimal frontmatter
with type: index and status: synthesis.
TODO-stub duplicate cleanup โ the dreaming pipeline and active-crawl jobs create index
entries as TODO placeholders (> **TODO**: Enrich this page). When real content is later
written, the index accumulates duplicate entries: one rich + one TODO stub for the same page.
During lint/dedup: (1) group index entries by wikilink target, (2) keep the rich entry
(first occurrence), (3) remove all TODO-stub duplicates. Exclude _index entries โ they
are legitimate per-directory. Automate with Python: scan for **- TODO** or Enrich this page
in index lines, match by wikilink target, remove duplicates bottom-up to preserve line numbers.
Triage candidate_wiki_path may point to wrong namespace or slug โ The triage pipeline generates candidate_wiki_path suggestions based on topic keywords, not entity/concept namespace awareness or existing slug conventions. Three failure modes:
concepts/X but entities/X exists (or vice versa). Checked by existing pitfall below.