메뉴
Create or enrich wiki entity and concept pages from raw articles, multi-source web research, newsletters, and tool/project investigations. Supports both article-driven enrichment and from-scratch entity page creation via website + docs + X/Twitter + GitHub/NPM research.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Comprehensive wiki health checking, maintenance, and remediation — duplicate detection, entity dedup/disambiguation, link auditing/fixing, page splitting, source-linking lint, wikilink remediation, tag taxonomy audit and normalization, pre-commit enforcement, language enforcement (JP→EN bulk translation, detection regex, cron-assisted migration), and decision-matrix-driven cleanup.
Comprehensive wiki maintenance: daily structural health checks (index reconciliation, log separator fixes, pipeline watchdog alerts), comparison page updates (adding items to multi-section comparison tables), and page relocation (moving/renaming pages while maintaining link integrity).
Pre-flight checklist and procedures for archiving, deleting, or migrating Hermes skills. Prevents accidental removal of cron-referenced skills. Covers the 3-layer skill structure and config.yaml management. Includes skill inventory management, promotion workflows, and archival conventions.
Karpathy's LLM Wiki: build/query interlinked markdown KB.
Deep analysis of blog authors' recent thoughts, philosophy, and positions. Goes beyond entity page creation to extract cited ideological positions, track thought evolution, and enable ongoing RSS monitoring for thought updates.
Query the blogwatcher-cli SQLite database for RSS scan results. Use pre-verified column names and query templates to avoid errors.
| name | wiki-entity-enrichment-from-article |
| description | Create or enrich wiki entity and concept pages from raw articles, multi-source web research, newsletters, and tool/project investigations. Supports both article-driven enrichment and from-scratch entity page creation via website + docs + X/Twitter + GitHub/NPM research. |
| trigger | When asked to explain an article, create concept pages from raw content, convert newsletter articles into wiki entries, ingest a lecture/workshop transcript (see references/transcript-ingestion.md), create a comprehensive entity page for a tool/project from scratch via multi-source research, create an AI model entity page from arXiv paper + project page + HuggingFace sources, enrich a status:skeleton entity page from scratch, batch-create company entity pages, batch-create person entity pages from X handles (see references/batch-person-entity-creation.md), ingest a conference/event summary article (see references/event-summary-article-workflow.md), ingest an X/Twitter post into the wiki (see references/x-note-tweet-ingestion.md), ingest a research paper with benchmark dataset (see references/research-paper-ingestion.md), OR cross-reference a newly ingested article with existing concept pages (connect/relate/link to PTC, RLM, SaC, etc.). |
When a user asks to "connect" or "relate" a newly ingested article to existing wiki concepts (e.g. "最近追加したPTC, RLMの概念とも関連つけて"), follow this workflow:
Users often reference concepts by abbreviation (PTC, SaC, RLM, DCI). Do NOT search by file glob — concept pages may have long names (programmatic-tool-calling) or live in unexpected locations. Instead:
search_files(pattern="PTC", target="content", path="wiki/index.md")
search_files(pattern="PTC", target="content", path="wiki/log.md")
The index.md and log.md are the authoritative lookup tables. Log.md often has richer context about what was added and when.
Read the concept pages to understand:
Add a section (e.g. "Beyond the Binary: Hybrid Architectures") that:
For each related concept page, add a one-line backlink in its Related Concepts section:
- [[concepts/new-article-concept]] — One-line description of the relationship
Also add related: frontmatter to the main concept page.
The article argued pipeline vs agent is binary. PTC, SaC, and RLM collapse that binary:
Each got a comparison table row and a one-line backlink. The main page got a "Beyond the Binary" synthesis section with an ASCII spectrum diagram.
For X/Twitter Note Tweet ingestion specifically, see references/x-note-tweet-ingestion.md for the full end-to-end pipeline (xurl fetch → raw article → entity page → related page updates).
⚠️ MANDATORY: Before any
write_file, runsearch_filesto check for existing pages. Overwriting 100-200+ line pages destroys curation work. Seereferences/pre-write-verification.mdfor the full protocol and recovery procedure.
Before creating a new page, determine whether the article describes a standalone concept or a sub-topic/extension of an existing concept: | A comparison or survey across multiple concepts | Create new concept page with cross-references | "Agent harness comparison" → new concept page |
Heads-up — Multi-target pattern for landmark articles: When a landmark article by a major figure reframes an existing concept (rather than introducing a new one), the correct action is almost always a three-part enrichment: (1) enrich the existing concept page with a dedicated section, (2) create an entity page for the author if they're not in the wiki yet, and (3) enrich any related entity pages that the article prominently references (e.g., the person who coined a term the article popularizes). This avoids the trap of creating a "new concept" page from content that is really a commentary on an already-documented concept.
| Article describes... | Action | Example |
|---|---|---|
| A new standalone concept, tool, person, or entity | Create new page in concepts/ or entities/ | New agent harness, new researcher |
| A major extension, feature, or sub-topic of an existing concept | Add a dedicated section to the existing concept page | MCP Apps → section in concepts/mcp.md |
| A landmark/commentary article about an existing concept by a major figure — article reframes/popularizes/situates an already-documented concept historically without introducing a standalone new concept | Enrich the existing concept page (add dedicated section) + create entity page for the author if not in wiki + enrich related entities that the article references | Tim O'Reilly's "End of Programming" about vibe coding → enrich vibe-coding page + create tim-oreilly entity + enrich addy-osmani with 70% Problem |
| A minor update or product announcement for an existing entity | Add a timeline entry or brief note to the existing entity page | New Claude version → add to entities/anthropic.md |
| A comparison or survey across multiple concepts | Create new concept page with cross-references | "Agent harness comparison" → new concept page |
Heuristic: If the new content would be confusing or incomplete without the parent concept's full context, it belongs as a section within the parent page. Standalone pages should be independently understandable.
When adding a section to an existing concept page:
tags and sources frontmatter[[raw/articles/...]] wiki-link to the raw articlewiki/raw/articles/<time> tags, article:published_time meta, search result snippets, Wayback Machine, RSS feed <pubDate>web_extract via JS rendering. Use old.reddit.com instead — prefix the URL with old. (e.g., old.reddit.com/r/LocalLLaMA/comments/...). This returns clean HTML with full post text and top comments.web_extract returns paywall overlay, try Jina Reader API (r.jina.ai/<URL>), sitemap.xml paths, or RSS feedsweb_extract returns only an AI-generated summary or truncated content:
browser_navigate to load the page first (the browser tool renders the full DOM)browser_console with document.body.innerText.substring() split into chunks of ~40,000 characters:
browser_console(expression="document.body.innerText.substring(0, 40000)")
browser_console(expression="document.body.innerText.substring(40000, 80000)")
<meta property="article:published_time"> or visible date text). These pages often have a date range (e.g., "2020-05-28–2022-01-02") — use the START dateweb_extract collapses the content. The text is available client-side in document.body.innerText. Split into chunks because browser_console may truncate very long strings (>100K chars)karpathy-writings, karpathy-ideas, karpathy-projects, karpathy-research), plan to distribute the article's content across ALL relevant sub-pages — not just the main entity page. The writings sub-page gets the post-mortem/details, the ideas sub-page gets the high-level theses/insights, and the concept page gets the case study. A single article from a prolific author may touch 3+ pages simultaneously.concepts/fine-tuning/pytorch-fsdp.md) that overlaps with the new page you're about to create. If the new page is richer and should be the canonical version, plan the promotion pattern (see Section: Subdirectory Promotion).---
title: "Concept Name"
created: YYYY-MM-DD
updated: YYYY-MM-DD
tags: [concept, tag1, tag2]
aliases: [alias1, alias2]
related: [[related-page-1]], [[related-page-2]]
sources: [URL or path to raw article]
---
# Concept Name
## Summary
Brief overview of the concept (2-3 sentences)
## Key Ideas
- Idea 1
- Idea 2
- Idea 3
## Terminology
- **Term**: Definition
## Examples/Applications
- Example 1
- Example 2
## Graph Structure Query (for knowledge base traversal)
When designing concept pages for a wiki knowledge base that supports graph traversal queries, add this section to define explicit, typed relationship edges. This enables structured queries like "find all concepts by author X" or "which concepts contrast with graphrag."
[this-concept] ──author──→ [entity: creator-name] [this-concept] ──contrasts──→ [concept: alternative-approach] [this-concept] ──extends──→ [concept: broader-framework] [this-concept] ──relates-to──→ [concept: adjacent-topic] [this-concept] ──embodies──→ [concept: underlying-philosophy]
**Edge types to consider:**
| Edge Type | When to Use |
|-----------|-------------|
| `author` / `coauthor` | Article author(s) as entity pages |
| `contrasts` | Directly opposing or competing approach |
| `extends` | The concept adds to or builds on a broader framework |
| `relates-to` | Adjacent topic with meaningful connection but no direct dependency |
| `embodies` | This concept is a concrete instance or application of a philosophy/framework |
| `teaches` | The concept functions as a tutorial or methodology guide |
| `precedes` / `follows` | Chronological ordering between related concepts |
| `part-of` | Hierarchical inclusion (sub-concept of a larger concept) |
Include actual wikilinks in a prose description below the diagram, e.g.:
> This section informs graph queries: authored by [[entities/hamel-husain]], contrasts with [[concepts/agentic-alternative-to-graphrag]], embodies [[concepts/harness-engineering]].
## Related Concepts
- [[related-page-1]]
- [[related-page-2]]
## Sources
- [Article Title](URL)
BEFORE creating any new entity file, run this pre-creation verification checklist. search_files alone is unreliable — it has false-negative issues where it returns 0 results for files that actually exist.
# 1. Filename search (most reliable — use this FIRST)
ls ~/wiki/entities/ | grep -i "partial-name\|@handle\|fullname"
find ~/wiki/ -name "*partial-name*" -o -name "*other-variant*" 2>/dev/null
# 2. Index check (catches alternate slugs and stubs)
grep -i "@handle" ~/wiki/index.md | head -5
grep -i "Full Name" ~/wiki/index.md | head -5
# 3. Content search for aliases (catches frontmatter aliases)
grep -ril "alias-name\|@handle" ~/wiki/entities/ --include="*.md" 2>/dev/null
# 4. Search under alternate slug patterns
# Known traps: GitHub handle vs real name (e.g., mitsuhiko vs armin-ronacher)
# handle vs full-name (e.g., vtrivedy10 vs varun-trivedy)
# first-last vs handle-first (e.g., zach-mueller vs the-zach-mueller)
for slug in "full-name-slug" "first-last" "@handle-stripped"; do
ls ~/wiki/entities/"$slug".md 2>/dev/null && echo "FOUND: $slug"
done
If ANY check finds a match, do NOT create a new file. Update the existing page instead (patch frontmatter, add sections, update sources). This applies even if the existing slug differs from what you planned (e.g., zach-mueller already exists even if you planned the-zach-mueller).
Duplicate cleanup workflow (if you discover mid-session that you created a duplicate):
rm ~/wiki/entities/your-duplicate-slug.mdrelated: frontmatter or inline wikilinks that reference the wrong slug — use patch on each affected pagepatchTwo variants — article-driven (has a raw article as starting point) and pure-web (no article, just URLs/handles):
| Variant | Starting Point | Target |
|---|---|---|
| Article-driven | Raw article exists but is insufficient alone | Conduct multi-source research to fill gaps |
| Pure-web | No article — just a website URL, GitHub repo, or tool name | Build entity page from zero via website scraping + docs + GitHub + X/Twitter + registries |
For pure-web (no article — tool discovered via website URL, GitHub, or X/Twitter mention):
web_extract the landing page) — identify the product category, pitch, supported platforms, key differentiatorweb_extract the docs root and 2-3 key subpages) — features, CLI reference, architecture, installationweb_search)web_search "ProjectName" arxiv and check the GitHub README's bottom sections (often links to papers). If found, scrape the arXiv abstract page (web_extract arxiv.org/abs/XXXX.XXXXX) as a key source for architecture details, benchmarks, and formal contributions. Add it to the entity page's sources and architecture/benchmark sections.Common sub-steps (both variants):
2. Scrape the website (web_extract the homepage and docs subpage) — extract features, architecture, supported agents, community links
3. Search X/Twitter — find the creator's pinned post / launch announcement (often contains key context like "v1 is live")
4. Search GitHub profile — discover related projects by the same creator (often builds an ecosystem)
5. Search NPM/Package registries — confirm package name, version, CLI command
7. Search for installation guides — third-party blogs may document setup steps and architecture
8. Create person entity page for the creator — If the creator (individual or team) doesn't have a wiki page yet, create one. The person entity page should include: professional background, related projects, their key theses/ideas, blog URL, social handles, and cross-references to the service/tool entity they created.
9. Check speaker profiles — conference speaker pages (Google Cloud Next, etc.) often contain professional bios
---
title: ProjectName
type: entity
created: YYYY-MM-DD
updated: YYYY-MM-DD
tags:
- devtools
- ai-agent
- orchestrator # adapt to project type
- open-source
aliases:
- alias1
- alias2
sources:
- raw/articles/article-file.md
- https://project-website.com/
- https://project-website.com/docs
---
# ProjectName
**ProjectName** is a [one-sentence description]. Built by [[entities/creator-name|Creator Name]], it provides [core value proposition].
[2-3 sentence overview of how it works — architecture principle, supported agents, key differentiator]
## Getting Started
```bash
npx @npm-package/name
Use H3 sections with bullet points for each feature. Prefer subsections over a flat list:
Description of what this feature does and why it matters.
...
| Agent | Status | Description |
|---|---|---|
| Agent A | Primary | Full integration |
| Agent B | Supported | Details |
Describe the architecture layers (client-server, hook-based, etc.) followed by an ASCII diagram:
┌──────────────────────────────┐
│ Browser UI (Port) │
└─────────┬────────────────────┘
│ WebSocket / REST
┌─────────▼────────────────────┐
│ Server (Node.js) │
│ Event Processing · State │
└──┬──────────┬──────────┬─────┘
▼ ▼ ▼
┌──────┐ ┌────────┐ ┌──────┐
│Agent A│ │Agent B │ │Agent C│
└──────┘ └────────┘ └──────┘
Built by Name (@handle), [role]. Also known for [other projects]. [Brief professional background paragraph from conference bios, LinkedIn, etc.]
### 5a. Concept Abstraction from Service/Tool Investigation
When investigating a service/tool that embodies a **broader paradigm or architectural concept** beyond the service itself, create separate concept pages and cross-link them:
**When to abstract:**
- The service is the **first/only canonical implementation** of a pattern worth documenting (e.g., searchcode.com → code-intelligence-for-llms, B2A)
- The service exemplifies a **named thesis or framework** from its creator (e.g., searchcode.com → "Marketing to the Machine" → B2A concept)
- Other services/projects could implement the same pattern (future-proofing the concept)
**Cross-referencing pattern:**
entity: service-example → "Canonical implementation of [[concept/p]]" concept: paradigm/concept → "Representative implementation: [[entities/service-example]]" entity: creator-person → "Author of the [[concept/paradigm]] thesis"
**Workflow order:** Raw article → entity page (service) → entity page (creator) → concept(s) → update entity page cross-references → index/log → commit
### 5b. Skeleton Entity Page Enrichment (From Scratch — No Article Source)
When a person/blogger entity page exists as a `status: skeleton` in `~/wiki/entities/` with no raw article to drive enrichment (just frontmatter + raw article references), enrich it from scratch via multi-channel web research:
1. **Read the existing page** — note the frontmatter (`title`, `url`, any references), current `status: skeleton`
2. **Check RSS feed for article inventory** — Try `/rss.xml`, `/feed.xml`, `/atom.xml` first. RSS gives all article URLs, titles, and dates in one call — much faster than scraping individual archive pages. Cross-reference RSS entries against the skeleton's reference list to map hash-based references (e.g., `mahadk.com--posts-arc-boost--1c8b9380`) to actual topics.
3. **Scrape the person's website/blog** (`web_extract` the about page, homepage, and 2-3 recent posts)
3. **Research professional background:**
- `web_search` for name + biography / founder / educator / researcher
- Check Wikipedia, LinkedIn, Crunchbase
- Check GitHub profile, Hugging Face profile, social media bios
4. **Identify core content themes:** What specific topics do they write about? (smolagents, LLM from scratch, fine-tuning, etc.)
5. **Search for specific article series** that demonstrate depth (e.g., "LLM from scratch part 1", multi-part tutorials, comparisons)
6. **Analyze writing style** from sampled blog posts
#### Target Page Structure (Person/Blogger Entities)
```markdown
---
title: slug
description: One-line description covering who they are + what they write about + why they matter
url: https://theirblog.com
type: entity
updated: YYYY-MM-DD
aliases: [handle1, Full Name]
tags:
- person
- ml-researcher
- educator
- python-developer
- [other-relevant-tags]
sources:
- https://theirblog.com/about
- [external-Bio-wiki-wikipedia-etc]
---
# Full Name
**Full Name** (handles: `handle`) is a [role descriptor]. [2-3 sentence intro covering background, major achievement, current focus]
## Overview
Detailed biographical paragraph covering:
- Programming/entrepreneurial background
- Notable projects or companies founded
- Current research/publishing focus
- Platform presence (GitHub, social media handles)
## Core Topics
Subsections for each major topic area they write about, with specific article titles and links:
### Topic Area 1 (e.g., smolagents Tutorials)
- **Article Title** (Month Year) — What it covers, why it matters
- **Article Title** (Month Year) — ...
### Topic Area 2 (e.g., LLM from Scratch Series)
Overview of the series scope, key experiments, model releases, and significant findings.
## Writing Style & Philosophy
Analysis of their writing approach — what makes their content distinctive. Characteristics to consider:
- Narrative voice (first-person, instructional, analytical)
- Transparency (sharing failures, uncertainties)
- Experimental grounding (code-driven claims)
- Comparative thinking (side-by-side evaluations)
- Breadth of perspective (cross-domain experience)
## [Entrepreneurial / Professional Background]
### Company/Project Name (YEAR–YEAR)
What they built, key milestones, acquisition/exit if any.
Use separate H3 subsections for each major venture.
## Cross-References
- **[[existing-entity-page]]** — Description of relationship and connection
- **smolagents** — (if no entity page exists yet, use plain text)
## References
- `references/mcp-apps-ingestion-2026-05-07.md` — Session reference for ingesting sub-topic articles as sections within existing parent concepts. Covers decision heuristic, workflow, and competing UI-over-MCP standards discovered during MCP Apps ingestion.
- `references/mcp-apps-session-2026-05-07.md` — Full session transcript for MCP Apps ingestion: decision to add as section within `concepts/mcp.md` rather than standalone page, comparison with competing UI-over-MCP standards, index/log update workflow.
All existing raw article references preserved from the original skeleton.
[[wikilink]] for existing pages, plain text for pages that don't exist yetstatus: skeleton entirelytype: entity (if missing)updated: YYYY-MM-DDtags array with relevant person/topic tagsaliases for handles and full namesources array with URLs to about page, Wikipedia, social profilesantirez-com.md / simon-willison.md depthWhen you discover that a subdirectory concept page (e.g., concepts/fine-tuning/pytorch-fsdp.md) already exists but your new top-level page is richer and should become the canonical reference:
search_files or ls reveals a .md file under concepts/subdirectory/ (e.g., concepts/fine-tuning/pytorch-fsdp.md)concepts/your-concept.md) with full content — this becomes the canonical reference> **Canonical page moved up.** This subdirectory page was merged into [[concepts/your-concept]] for the comprehensive reference. Key content retained below.updated date, add aliases if needed, add the new top-level page to related:, add the new raw article to sources:related: frontmatter to include concepts/fsdp-qlora, concepts/qlora, and the new top-level pagegrep -r "concepts/fine-tuning/pytorch-fsdp" ~/wiki/ --include="*.md" | grep -v ".md.swo\|.git"
concepts/_index.md if the subdirectory has an index page — add a note pointing to the new top-level pagewiki/index.md — add an entry for the new top-level page in the Concepts section (simple - prefix format, no renumbering needed)related: frontmatter on BOTH pages (bidirectional cross-references)index.md for the subdirectory page's existing entry — update its description to note the redirect, don't remove itFrom this session: concepts/fine-tuning/pytorch-fsdp.md (100 lines, Axolotl-focused) → concepts/pytorch-fsdp.md (239 lines, comprehensive). Subdirectory page updated with redirect notice, new frontmatter fields, and cross-refs to the new top-level page.
When the user provides an arXiv paper + official project page (no blog article) and asks to create a wiki page for an AI model/system:
Source triad: arXiv paper → project page → HuggingFace model card. These three sources together provide architecture details, benchmarks, and deployment info that a single blog post wouldn't.
Research steps (modify section 5's general steps for this pattern):
web_extract the arXiv abstract page → paper summary (architecture, innovations, benchmarks)web_extract the official project page → capabilities, features, demo linksweb_extract the HuggingFace model card → model variants, training details, safety features, licenseweb_search the model name for third-party analysis (benchmarks against competitors, community reception)Entity page structure for AI models (extends section 5's tool/project template):
---
title: "Org ModelName"
type: entity
created: YYYY-MM-DD
updated: YYYY-MM-DD
tags:
- entity
- model
- [org-name]
- tts / llm / speech / vision # adapt to model type
- diffusion / transformer / mamba # adapt to architecture
- open-source / proprietary
related:
- [competitor-entity]
- [broader-concept-page]
sources:
- raw/articles/paper-summary.md
- https://arxiv.org/abs/XXXX.XXXXX
- https://org.github.io/ModelName/
- https://huggingface.co/org/ModelName
status: skeleton # optional — remove after enrichment
---
# Org ModelName
## Architecture
Break down into 2-3 core components with parameters:
### Component 1 (e.g., Tokenizer)
- Base technique (σ-VAE, flow-matching, etc.)
- Key specs: frame rate, compression ratio, parameter count
- Novelty vs prior art (e.g., "80× better than Encodec")
### Component 2 (e.g., Language Model)
- Base model / size
- Context length and training curriculum
- Role in the pipeline
### Component 3 (e.g., Decoder/Head)
- Layer count, parameter count
- Inference technique (DDPM, CFG, DPM-Solver)
- Latency characteristics
## Model Variants
| Variant | Base LLM | Params | Capability | Max Duration | Latency | Status |
|---------|----------|--------|-----------|-------------|---------|--------|
| Variant-A | ... | ... | Full | ... | ... | Released |
| Variant-B | ... | ... | Streaming | ... | ~Xms | Released |
## Key Capabilities
| Capability | Detail |
|-----------|--------|
| Max duration | ... |
| Max speakers / inputs | ... |
| Languages | ... |
| License | ... |
## Benchmark Performance
- Conference venue (e.g., ICLR 2026 Oral)
- Objective metrics vs competitors (WER, similarity, etc.)
- Subjective human evaluation results
## Safety & Limitations
### Safety Features
- Content filtering, watermarking, logging, disclaimers
### Out-of-Scope Uses
- Voice impersonation, real-time deep-fakes, etc.
### Known Limitations
- What the model cannot do (no overlapping speech, single language only, etc.)
## Comparison: ModelName vs CompetitorModel
| Aspect | This Model | Competitor |
|--------|-----------|-----------|
| Architecture | ... | ... |
| Size | ... | ... |
| Max output | ... | ... |
| Languages | ... | ... |
| License | ... | ... |
## Related Concepts
- [[entities/competitor-model]]
- [[concepts/broader-category]]
## TODO
- [ ] Monitor for updates
- [ ] Enrich with hands-on evaluation
</markdown>
**Key sections unique to model entities** (not in tool/project template):
- Architecture breakdown with parameters (tokenizer, LLM, decoder)
- Model variant table (sizes, capabilities, availability)
- Benchmark performance with venue (ICLR, NeurIPS)
- Safety & Limitations section (guardrails, out-of-scope uses)
- Comparison table with similar models (architecture-level, not just feature-level)
**Concept page competitive positioning update** (AFTER creating the entity):
This is a critical step that differs from tool/project ingestion:
1. Search for **broader concept pages** that categorize this type of model:
```bash
grep -r "competitor-name\|similar-model" ~/ai-topics/wiki/concepts/ | grep -oP '\[\[concepts/[^\]]+' | sort -u
sources: frontmatter if appropriateconcepts/speech.md or the most general parent concept:
[[entities/new-model]] to the Related Concepts listTool/project entity pages require three index updates:
Pitfall — stubs missing from index.md: A concept or entity page may exist as a file (possibly created by an earlier cron job or bulk import) but be absent from wiki/index.md. Always verify the page appears in the index after creation or enrichment. If missing, add an entry in the alphabetically-correct location with a one-line summary.
wiki/entities/_index.md — Add entry with wikilink and one-line summary. Verify alphabetical sort order with read_file before patching.wiki/index.md (top-level wiki index) — Add entry under Entities section with wikilink and shorter summary. Same alphabetical placement.wiki/log.md — Prepend entry with date, entity wikilink, key features summary, and raw article path.When patching entities/_index.md, concepts/_index.md, or wiki/index.md:
Read the surrounding lines first with read_file(offset, limit) to find the exact insertion point
Use patch with enough context to make the old_string unique (include surrounding lines)
Verify the patch result by reading again — check for formatting issues (e.g., |- instead of -, or ||- double-pipe)
PIPE PREFIX PITFALL: The patch tool is fragile with pipe-prefixed list entries that appear in entities/_index.md and concepts/_index.md. When these files use |- prefix format (because of markdown table syntax), and your old_string uses - (without the pipe), the patch tool adds an extra | on every inserted line creating ||- artifacts. Fix: After every patch on _index.md files, immediately read_file the affected lines. If you see ||- artifacts, use sed -i 'N,Ns/^|//' PATH to strip the extra leading pipe. Then re-verify.
PARTIAL-MATCH CORRUPTION on patch: When your old_string matches only a PREFIX of the target line (not its full content), the patch tool replaces the matched portion and appends the REMAINING original text onto your new_string. This creates corrupted entries where original line tail leaks into the new content. Fix: Always include enough trailing context in old_string to uniquely identify the ENTIRE line — read the file with read_file(offset, limit) and copy the exact bytes. After every patch on index files, immediately re-read the affected lines to detect appended garbage. If found, fix with a second patch that replaces the corrupted substring.
EMBEDDED LINE-NUMBER CORRUPTION in _index.md and index.md files: Several index files (entities/_index.md, concepts/_index.md, AND the main wiki/index.md) have old line numbers embedded in the actual file content (e.g., 169|- [[entities/jaya-gupta]] where 169 is an embedded line number, not a read_file display artifact — it is literally in the file). The patch tool consistently fails on these files — it either cannot find a match or produces hybrids like 171| 169|- [[slug]]. Additionally, the embedded number format varies: some lines have NNN|- prefix, some have - prefix, causing unpredictable patch behavior. DO NOT use patch for any index files that show embedded line numbers in read_file output. Always verify with sed -n 'N,Np' PATH to see the actual file bytes. Use Python for ALL insertions instead:
The three index files use DIFFERENT formats depending on section. Accidentally applying the wrong method will corrupt the file.
| File / Section | Format | Example | Insertion Method |
|---|---|---|---|
Entities section in wiki/index.md | `NNN | - [[slug]]` (embedded line numbers + pipe-dash) | ` 256 |
Concepts section in wiki/index.md | - [[slug]] (simple dash-space, NO line numbers) | - [[concepts/accelerate]] | Simple patch or list.insert() — no renumbering |
| entities/_index.md | `NNN | - [[slug]]` (embedded line numbers) | Same as Entities section |
| concepts/_index.md | `NNN | - [[slug]]` (embedded line numbers) | Same |
Entities section =
NNN|-format → use Python with renumbering. Concepts section =-format → use simplepatchorinsert.
To verify which format a section uses before editing:
# Check a few lines around your insertion target
sed -n '200,210p' ~/wiki/index.md | head -5 # Check format
If you see NNN|- anywhere near your target, use Python.
import re
with open('/opt/data/wiki/entities/_index.md', 'r') as f:
lines = f.readlines()
# Find the insertion point by matching existing entries
before_idx = None
after_idx = None
for i, line in enumerate(lines):
if 'thorsten-ball' in line: # <-- adjust to your anchor entry
before_idx = i
if 'tim-sh' in line: # <-- adjust to next-after entry
after_idx = i
# Extract the embedded line number format from existing lines
match = re.match(r'(\s*)(\d+)(\|)', lines[before_idx])
indent = match.group(1) # leading spaces
base_num = int(match.group(2)) # embedded line number
# Create the new line (increment line number by 1)
new_line = f'{indent}{base_num + 1:3d}|- [[slug]] — One-line description.\n'
lines.insert(after_idx, new_line)
# Renumber all lines from before_idx onward
current = base_num
for i in range(before_idx, len(lines)):
m = re.match(r'^(\s*)(\d+)(\|)', lines[i])
if m:
prefix, num, sep = m.group(1), int(m.group(2)), m.group(3)
# Only renumber lines with the same indent pattern
if m.group(1) == indent and num >= base_num:
lines[i] = f'{prefix}{current:3d}{sep}{lines[i][m.end():]}'
current += 1
with open('/opt/data/wiki/entities/_index.md', 'w') as f:
f.writelines(lines)
Key patterns in this approach:
'slug' in line to find anchor entries by content (reliable across sessions)indent + line_num + |- ` prefix + slug + descriptionIf you need to patch a corrupted line instead:
Do NOT use sed -i 'N,Ns/^|//' — this strips the leading artifact but leaves content malformed. Use Python:
with open('PATH', 'r') as f:
lines = f.readlines()
lines[idx] = f' {old_line_number:3d}|- [[slug]] — Description\n'
with open('PATH', 'w') as f:
f.writelines(lines)
When creating an entity page for an organization, research group, or company from multi-source research, you may discover hidden connections to technologies that were previously documented from a different angle. The entity may have co-developed a technique that already exists as a concept page — but neither the entity page nor the concept page cross-references the other.
This happens when:
Example from this session: Creating [[entities/mobius-labs]] revealed they were a co-developer of FSDP+Q-LoRA (Answer.AI × Tim Dettmers × Hugging Face × Mobius Labs). The existing [[concepts/fsdp-qlora]] page credited "Answer.AI, Tim Dettmers, and Hugging Face" — Mobius Labs was missing from the cross-reference because none of the prior sources focused on them.
# Search concept pages for the collaboration name
grep -r "Mobius\|HQQ\|FSDP.*QLoRA" ~/wiki/concepts/ --include="*.md" | head -20
# Search existing concept pages for the entity name
grep -r "mobius-labs\|Mobius Labs" ~/wiki/concepts/ --include="*.md" | head -20
# Search raw articles for mentions
grep -r "Mobius Labs" ~/wiki/raw/articles/ --include="*.md" | head -20
related: frontmatter AND add a cross-reference section (e.g., "Key People & Organizations" or a note in "Related Concepts")related: frontmatter with a wikilinksources: frontmatter[[entities/new-entity]] as a note in the raw article's tags or a commentpatch (never write_file) to preserve existing contentIf a bulk-processing record (e.g., ~/wiki/bulk-processing/bulk-*.md) lists this entity as ❌ ファイル未作成, update it to ✅ 作成済み after creating the page.
status: skeleton — fresh entity pages based on multi-source research are complete, not skeletons. Only use status: skeleton for placeholder pages created during bulk processing or automatic blog/X account scraping.type: entity (not type: person or type: concept)tags from SCHEMA.md taxonomy: devtools, ai-agent, orchestrator, open-source, product, coding-agents, etc.sources array must include both raw article paths and external URLs (website, docs, etc.)cd ~/ai-topics
git add wiki/
git commit -m "wiki: <summary>"
git push
wiki/entities/antirez-com.md or wiki/entities/simon-willison.md[[links]] must resolve to existing pageswiki/concepts/ firstindex.md AND log.mdThe pre-commit hook validates ALL staged files, not just yours. If cron jobs (active-crawl, blog-wiki-ingest, etc.) created new concept pages with tags not yet in SCHEMA.md, your commit will be blocked even though your own changes are clean.
Workflow when commit fails with tag violations:
patch() on the appropriate category linegit add wiki/ && git commit -m "..."This is NOT an --no-verify situation. The tags are valid; they just need to be registered. Adding them to SCHEMA.md is the correct fix and takes ~30 seconds per batch.
When the raw article already exists in wiki/raw/articles/ and a concept page already references it, the entity page enrichment checklist is:
sources frontmatter YAML list[[wikilink]] to concept pageupdated date to todayupdated date is stale, bump it toolog.md — prepend entry (file may be large; use terminal with sed -i to insert after line 3)index.md validation passes, then commit and pushlog.md grows continuously and can exceed 100KB. The patch() tool requires reading the full file first, which fails for very large files. Use terminal with sed -i to prepend entries:
sed -i '4i\## [date] entry title\n\nContent here\n\n' log.md
Line 4 is typically right after the header (lines 1-3: title, description, blank line).
ls ~/wiki/entities/ | grep -i "partial-name" — manual filename scan (most reliable)find ~/wiki/ -name "*partial-name*" 2>/dev/null — broad filesystem searchgrep -ril "@handle\|Full Name" ~/wiki/entities/ — content/alias searchmitsuhiko vs armin-ronacher), slug variants (e.g., zach-mueller vs the-zach-mueller). If an existing page is found under ANY slug, update that page — never create a new file.index.md, always verify the section header first (Concepts vs Entities vs Comparisons) before patching. Each insertion needs its own patch call at the correct alphabetical position./opt/data/home/wiki/ instead of the canonical ~/wiki/ (which resolves to /opt/data/wiki/ → /opt/data/ai-topics/wiki/). After delegation, always verify files landed in the git repo: ls /opt/data/ai-topics/wiki/entities/NAME.md and ls /opt/data/ai-topics/wiki/concepts/NAME.md. If missing, copy from the stale path: cp /opt/data/home/wiki/entities/NAME.md /opt/data/ai-topics/wiki/entities/ and add to git.index.md, read the full file with offset/limit to find the correct insertion line for each entry, then apply patch for each one individually. Never assume alphabetical order without verifying.patch may error with Escape-drift detected. This happens when the exact bytes in the file don't match what the tool serialized. The recommended verbatim re-read fix often still fails because Unicode gets re-serialized differently on each read. Two fallback approaches (try in this order):
old_string to only the unique line(s) that lack the problematic characters, avoiding smart quotes, &, em-dashes, and CJK. For example, if line 38 has smart quotes but line 37 is clean, write your patch to match just line 37 + the start of line 38. This is the fastest fix and preserves the patch workflow.sed — If shortening fails, use sed for line insertion: sed -n 'N,Np' PATH to check line numbers, then sed -i 'Ni|- [[entities/slug]] -- Description' PATH. Always verify with sed -n 'M,Np' afterward.\u201c/\u201d/\u2019/\u2014):
\u2019 as a literal backslash-escaped sequence in new_string writes the text \u2019 into the file instead of the intended Unicode character. Fix: Always use plain ASCII equivalents instead: ' (not \u2019/right single quote), " (not \u201c/\u201d), -- (not \u2014/em-dash). Re-read the surrounding lines after every patch that touches a file with smart quotes to verify no escaped literals leaked in.wiki/index.md, entities/_index.md, concepts/_index.md) may use variable prefix formats — some lines use - (dash-space), others use NNN|- (embedded line number + pipe-dash), and the format can change mid-file. Never assume a uniform prefix. Always verify with sed -n 'N,Np' PATH before patching to see the actual bytes. If ANY line in the target section has embedded line numbers, use the Python approach instead of patch (see EMBEDDED LINE-NUMBER CORRUPTION above).vibe-coding exists under BOTH harness-engineering/agentic-workflows/vibe-coding AND as a top-level vibe-coding entry), patch matches the first occurrence — which may be in the wrong section. This can silently delete unrelated lines (e.g., context-engineering was dropped when the patch targeted the top-level listing but matched the harness-engineering subdirectory). Fix: Before patching, always read_file(offset=X, limit=5) at the intended target to get the EXACT surrounding context. Include 1-2 lines of surrounding context in old_string to disambiguate which occurrence you're targeting. If the slug is not unique, use more surrounding context or switch to a Python insertion.list.insert() then doesn't shift unprocessed insertion points). Alternatively, insert one entry, re-read the file, then insert the next. The descending-sort approach is simpler and works for any number of insertions.concepts/_index.md file does NOT list all concept pages (many like mismanaged-geniuses-hypothesis exist in filesystem but are absent from _index.md). Do NOT assume every new concept needs an _index.md entry. Only add entries there when updating a pre-existing bucketed section or when the pattern is clearly comprehensive. The main wiki/index.md IS the authoritative catalog -- always update that one.When a concept page is already referenced in the wiki (via wikilinks in entity pages, related: frontmatter, or an existing index.md entry) but the actual .md file doesn't exist:
(Full section unchanged — see above)
When an existing concept page covers dimension A (e.g., pretraining) and a new article covers dimension B (e.g., RLHF/alignment) on the same umbrella topic, enrich by adding parallel H2 sections rather than merging or creating a new page. See references/multi-dimensional-concept-enrichment.md for the full workflow, detection signals, and comparison with case studies/multi-paper expansion.
concepts/<slug>/ directory but no root pageFalse-negative pitfall with search_files (target="files"): The search_files tool can return 0 results for exact-name patterns even when the file exists -- glob/regex matching is unreliable. Always cross-verify suspected-non-existent files with a terminal command: ls ~/wiki/concepts/SLUG.md 2>/dev/null || echo NOT FOUND. If the file exists despite search_files returning 0, your task shifts from backfill to enrichment -- skip the backfill workflow and instead patch the existing file to add the content that the references expected.
concepts/slug/sub-page.md exists, the root page should reference them and structure itself as an umbrella.patch.related: frontmatter — the wikilink now resolves, so verify any linked descriptions still make sense.concepts/harness-engineering was referenced in entities/hamel-husain.md (related: section), had sub-pages under concepts/harness-engineering/agentic-engineering/, and had an index.md entry — but no concepts/harness-engineering.md existed. The article "The Revenge of the Data Scientist" served as the foundational source to backfill it.
X article URLs often return HTTP 500 (auth wall) OR succeed but contain only metadata (title, author, description) with no body content.
When scraping fails or returns incomplete content:
$GETXAPI_KEY is set in the environment, fetch the article body via:
curl -s -H "Authorization: Bearer <GETXAPI_KEY>" \
"https://api.getxapi.com/twitter/tweet/article?id=<TWEET_ID>"
x.com/user/status/NNN), NOT the article entity ID. See [[social-media/x-article-getxapi-fallback]] for response structure and markdown conversion. If successful, save as full-content article with getxapi: true frontmatter and skip the mirror-search steps below.article.title field in the tweet response, NOT from trying to xurl read <article_id> which returns "Could not find tweet with id" — X Article IDs are not tweet IDs)web_search with: "<article title>" author "<handle>" site:substack.com or "<article title>" "<author name>"site:twitter.com "<article title>" (may find threads summarizing the article)source_fallback: trueWhen NO mirror exists but the article title points to a well-known author + known technical domain, you can reconstruct meaningful wiki content through multi-source synthesis:
data.article.title.When a real-world article describes how a company/product used a technology:
llm-as-judge to dspy.md)sources: array in frontmatterllm-as-judge.md or related pages if the case study demonstrates evaluation patternspatch to insert, not write_file (preserves existing content)read_file after patchingExample structure for a case study insertion:
### Company X Use Case (Month Year)
Company X used TECHNOLOGY to solve PROBLEM.
**Three-stage approach:**
| Stage | Model | Technique | Result |
|-------|-------|-----------|--------|
| A | model-v1 | method-A | metric-A |
| B | model-v2 | method-B | metric-B |
**Key patterns:**
- Pattern 1: description
- Pattern 2: description
[Source: Company X Blog](URL)
[Source: Company X Blog](URL)
## Single-Experiment Multi-Page Distribution (★★☆☆☆)
When one article/blog post describes a single experiment with quantitative results that validates an existing concept, enrich multiple existing pages simultaneously at different granularity levels rather than creating a new page.
### Detection
- Article describes one controlled experiment with metrics (benchmarks, ablation, methodology)
- The experiment validates a named hypothesis or concept that already has a wiki page
- The same results are relevant to entity pages (author), sibling concept pages (technical mechanism), and sub-entity pages (project-specific benchmarks)
### Workflow: Descending Granularity
Distribute the experiment data across pages from most-general to most-specific:
| Level | Target | What to add | Example from this session |
|-------|--------|-------------|---------------------------|
| 1. Parent concept | The validated hypothesis/framework page | Full experiment section: methodology, results table, ablation, connection to thesis | MGH page: LongCoT Experiment with 3 failure modes, ablation proof |
| 2. Sibling concept | The technical mechanism page | Experiment section alongside parallel experiments on same benchmark, call out differences in approach | RLM page: Zhang LongCoT experiment positioned alongside Weitekamp results |
| 3. Author entity | The person page | Blog post entry + one-line proof bullet under relevant section | Alex Zhang: new row in Blog Posts table + proof bullet under MGH |
| 4. Sub-entity | The project sub-page | One-line benchmark result in the benchmark list | Omar Khattab RLM sub-page: LongCoT-mini 65.6% in benchmark list |
**Key rules:**
- Each page gets a DIFFERENT amount of detail: parent concept = fullest (methodology + results + interpretation), sibling concept = mid (results table + key insights), entity = minimal (blog entry + one bullet), sub-entity = one-liner (benchmark number)
- No new pages are created -- all enrichment is patch on existing pages
- The same raw article serves as source for all patches
- Always cross-reference between levels in the sibling concept section (e.g., See full analysis in [[concepts/parent-concept]])
## Multi-Part Series Incremental Expansion (★★★★☆)
When the user progressively adds individual parts/articles from a known **multi-part series** (blog series, talk series, video series, paper series) where a parent series overview page acts as a hub connecting all parts.
### Detection
- User says "I added [part N] from this series, also add [part M]"
- The new URLs follow a predictable pattern (`series-name/p2-xxx`, `series-name/p3-yyy`)
- The parent series page already exists as a concept page listing all parts
- Each part has its own distinct presenter/author with a specific concept contribution
### URL Discovery for Paywalled/Subscription Sites
When a blog is on a subscription platform (Ghost, Substack, etc.) that blocks homepage scraping with a paywall overlay:
1. **RSS first** — check `/feed.xml`, `/rss.xml`, `/atom.xml`
2. **If no RSS** — check the **sitemap**: `/sitemap.xml`, `/sitemap-posts.xml`, `/sitemaps.xml`. Ghost sites auto-generate `sitemap-posts.xml` with ALL post URLs and lastmod dates.
3. **Filter by series** — grep/parse the sitemap for the series URL pattern (e.g., `automation-series-`) to extract all parts at once
4. **Compare with sitemap lastmod** — the sitemap's `<lastmod>` field reveals whether parts are published in chronological order, useful for establishing series sequence
5. **Avoid homepage scraping** — subscription sites return only signup UI from the homepage, making `web_extract` useless for navigation
### Workflow
**First part of a new series:**
1. Create the **parent series overview concept page** — covers all parts with a table, key insights, and sources for the overview article
2. Create **individual sub-concept page** for the first part — captures the unique concept contribution of that part
3. Update entity pages for key people (host, presenters)
4. Set up the graph structure query with `part-of` and `includes` edges
**Each subsequent part:**
1. Save raw article
2. Create **new sub-concept page** for this part — independent concept page with its own aliases, sources, and graph edges
3. **Update the parent series concept page** — add the new part to the Parts table, Key Insights table, and Part Details section
4. Update entity pages for the presenter if needed (timeline entries, sources)
5. Cross-link the new sub-concept page to the series page and sibling parts
6. If a person entity is mentioned but hasn't been created yet, check for existing pages under different slugs first (see Batch Creation > Person Extraction)
7. Update index.md, log.md, and commit
### Key Differences from Multi-Paper Expansion (single concept)
| Aspect | Multi-Paper (single concept) | Series Expansion (hub + sub-pages) |
|--------|------------------------------|-------------------------------------|
| Page structure | Multiple sources → ONE concept page | Each part → OWN concept page + parent hub |
| Source relationship | Perspectives merged into levels | Parts are independent siblings under a parent |
| Cross-referencing | Internal level references | `part-of` / `includes` graph edges |
| Entity updates | Optional | Likely needed (host + presenters) |
| Index updates | May skip if description unchanged | Always needed (new sub-concept pages) |
### Pitfalls
- **Check for existing entity pages before creating** — the presenter may already have a page under a different slug (e.g., `benjamin-clavie` vs `ben-clavie`). Search filenames and content before creating duplicate entity pages.
- **Fix stale ben-clavie/benjamin-clavie type refs** — when editing the parent series page, check that all wikilinks point to the correct existing slug. The initial creation may have used a slug that doesn't match an existing entity page.
- **Don't rewrite the entire parent page** — only patch new sections in (Part Details, tables, sources). The parent page grows incrementally with each addition.
- **Re-read parent page before each addition** — structure may have changed since the previous part was added.
- **Audit existing wikilinks on the parent page** — when re-reading the parent page, check that all existing wikilinks (especially for earlier parts and related concepts) point to correct existing slugs. The initial creation may contain stale links (e.g., `context-graph` instead of `context-rot`). Fix them as part of the same patch session — don't defer to a separate cleanup run.
- **Index page count** — each new sub-concept page increases the count by 1. Track cumulative additions across the session.
- **Document unique intellectual style when presenters use distinctive terminology** — Some presenters (e.g., Bryan Bischof with "The Map is Not the Territory" / "Curving Space" / buzzword deconstruction, Kelly Hong with "Context Rot") bring a distinctive philosophical vocabulary. When detected, create a dedicated subsection in the concept page that:
1. Labels the named metaphor/framework explicitly (e.g., `**"Curving Space"** — intention...`)
2. Traces its intellectual lineage if applicable (e.g., Korzybski for map/territory, physics metaphors for curving space)
3. Explains the **rhetorical intent** — what problem does this framing solve? (e.g., reframing index engineering from passive to active, stripping marketing language down to pipeline operations)
4. Quotes the presenter's exact phrasing with context
[Source: Company X Blog](URL)
## Single-Experiment Multi-Page Distribution (★★☆☆☆)
When one article/blog post describes a single experiment with quantitative results that validates an existing concept, enrich multiple existing pages simultaneously at different granularity levels rather than creating a new page.
### Detection
- Article describes one controlled experiment with metrics (benchmarks, ablation, methodology)
- The experiment validates a named hypothesis or concept that already has a wiki page
- The same results are relevant to entity pages (author), sibling concept pages (technical mechanism), and sub-entity pages (project-specific benchmarks)
### Workflow: Descending Granularity
Distribute the experiment data across pages from most-general to most-specific:
| Level | Target | What to add | Example from this session |
|-------|--------|-------------|---------------------------|
| 1. Parent concept | The validated hypothesis/framework page | Full experiment section: methodology, results table, ablation, connection to thesis | MGH page: LongCoT Experiment with 3 failure modes, ablation proof |
| 2. Sibling concept | The technical mechanism page | Experiment section alongside parallel experiments on same benchmark, call out differences in approach | RLM page: Zhang LongCoT experiment positioned alongside Weitekamp results |
| 3. Author entity | The person page | Blog post entry + one-line proof bullet under relevant section | Alex Zhang: new row in Blog Posts table + proof bullet under MGH |
| 4. Sub-entity | The project sub-page | One-line benchmark result in the benchmark list | Omar Khattab RLM sub-page: LongCoT-mini 65.6% in benchmark list |
**Key rules:**
- Each page gets a DIFFERENT amount of detail: parent concept = fullest (methodology + results + interpretation), sibling concept = mid (results table + key insights), entity = minimal (blog entry + one bullet), sub-entity = one-liner (benchmark number)
- No new pages are created -- all enrichment is patch on existing pages
- The same raw article serves as source for all patches
- Always cross-reference between levels in the sibling concept section (e.g., See full analysis in [[concepts/parent-concept]])
## Multi-Part Series Incremental Expansion (★★★★☆)
When multiple research papers, technical reports, or blog posts share a common dataset/benchmark/context and are added to the SAME concept page over multiple separate user requests.
### Detection
- A user shares a paper or report about a topic you've already expanded
- The new source references the **same benchmark** (e.g., BrowseComp-Plus, Oolong, NQ)
- The new source presents a **complementary or contrasting approach** to existing content on the same concept page
### Organization Pattern: Multi-Level Framework
When 3+ sources cover the same topic from different angles, organize them into numbered levels or perspectives:
Each level should be **self-contained** (readable standalone) but explicitly cross-reference the other levels:
- "This directly challenges the assumption from Level 1 that better retrievers are always beneficial."
- "See Level 3 for a contrasting approach using file-system tools."
### Incremental Update Checklist
For each new source added to an existing concept page:
1. **Read the full current concept page** — understand existing structure and find correct insertion point
2. **Create appropriate section** — H3 subsection under existing level, or new H2 level if it's a genuinely new paradigm
3. **Add comparison tables** where the new source contrasts with existing content (shared benchmarks make direct comparisons valuable)
4. **Update frontmatter**: bump `updated` date, add new `aliases`, new `sources` (both raw article/paper path and external URL)
5. **Update the `tags` array** — add new relevant tags (e.g., `rl-training` for RL-trained retrieval)
6. **Patch the Sources section** at the bottom — prepend new entry
7. **Update `index.md`** only if the concept page description changes significantly (not for every incremental expansion)
8. **Update `log.md`** — prepend new entry with source name, key findings, raw article path, and URL
9. **Commit each incrementally** — `git commit -m "wiki: add <source-name> to <concept-page>"`
10. When a source is arXiv-only with no peer-review venue, note `(arXiv-only, ingested per user request)` in log.md
### Pitfalls
- Don't rewrite the entire page — only patch the new section in. The user may add more sources later.
- When adding a contrasting perspective, preserve existing content verbatim. Add new sections, don't modify old ones unless fixing errors.
- If the new source comes from a company blog/tech report (not arXiv/peer-reviewed), save to `raw/articles/`. Only save to `raw/papers/` for arXiv or arXiv-like preprints.
- Always re-read the page before the second incremental addition — the structure may have changed since your first addition.
- Check for emerging structural patterns: when 2 sources exist, a third may complete a framework (e.g., IR vs File-System vs RL-Trained becomes a clean 3-level model). Design the structure with future expansion in mind.
- **Cross-link successive additions**: When adding source N+1, explicitly reference previously-added sources within the same page (e.g., "This mirrors the Level 1 finding (SID-1...)"). This turns a pile of sources into a coherent, cross-referenced framework.
- **Production implementation validates theory**: When adding a production/real-world source to a multi-level concept page, check if it validates ALL levels simultaneously. Call out each level's correspondence explicitly with a "Connection to the X-Level Framework" subsection. This is distinct from a normal case study (which validates one thing) — you need to enumerate each level separately.
|- **arXiv-only user override**: When the user explicitly requests an arXiv-only paper with no peer-review venue, note `(arXiv-only, ingested per user request)` in log.md AND add `status: blocked` + `blocked_reason: no_peer_review` in the raw paper's frontmatter. Run normal concept enrichment; the paper isn't cited as a primary source but its findings enrich the concept page.
|- **Add validated patterns table when 3+ sources converge**: After integrating 3+ sources, append a `#### Validated Patterns` table at the end of the newest section. Each row maps a finding back to the level/source that established it (e.g., | BM25 > embeddings for agents | Level 1: Query Mismatch). Serves as quick-reference summary of what each source contributed and how they interrelate.
|- **Frontmatter source dedup after each patch**: When patching the `sources:` frontmatter array multiple times, duplicate entries can accumulate silently. After each frontmatter patch, `read_file` to verify no duplicates exist. Clean up with `patch(replace_all=True)` if found.
## Blog Feed Discovery (RSS absent → sitemap.xml fallback)
When adding a blog to `blogwatcher-cli`, **verify RSS/Atom existence first** before adding:
1. Try common feed paths: `/feed`, `/feed.xml`, `/rss.xml`, `/atom.xml`, `/blog/feed`, `/blog?format=rss`
2. Use `curl -s -o /dev/null -w "%{http_code}" <url>` or Python `urllib.request.urlopen()` in `execute_code`
3. Check HTML `<head>` for `<link type="application/rss+xml">` or `<link type="application/atom+xml">` tags
**If RSS/Atom found** → Add with: `blogwatcher-cli add "Blog Name" https://example.com` (auto-discovers feed).
**If ALL feed paths return 404/empty AND no feed link in HTML** → Blog has **no RSS feed**. Do NOT rely on blogwatcher-cli scanning (it reports "No feed or scraper configured"). Instead:
1. **Try sitemap.xml**: Check `/sitemap.xml` and common variations (`/sitemaps.xml`, `/sitemap_index.xml`). Many static-site blogs (Astro, Hugo, Jekyll, Next.js) auto-generate sitemaps even without RSS feeds.
```python
# Python snippet for probing
import urllib.request, urllib.error, xml.etree.ElementTree as ET
url = "https://example.com/sitemap.xml"
try:
resp = urllib.request.urlopen(url, timeout=10)
if resp.status == 200:
tree = ET.parse(resp)
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
locs = tree.findall(".//sm:loc", ns)
blog_urls = [l.text for l in locs if "/blog/" in (l.text or "")]
print(f"Found {len(blog_urls)} blog posts in sitemap")
except: print("No sitemap")
blogwatcher-cli add "Blog Name" https://example.com. This registers it for tracking even though periodic RSS scans won't find articles. The entity page can note "no RSS feed."web_extract(). Save each as wiki/raw/articles/YYYY-MM-DD_source-slug.md.entities/org-name.md) and flagship project entity (entities/project-name.md) simultaneously — the blog likely covers both with cross-links between them.Framer-hosted blogs (detectable by "Powered by Framer" footer or framer.app references) use client-side dynamic routing: the blog index page displays article titles/URLs, but individual article URLs return 404 when fetched server-side by web_extract(). This is NOT the same as a paywall or access control — Framer simply doesn't SSR individual blog posts.
Primary technique: Browser-based URL discovery (use this FIRST, most reliable):
browser_navigate(url="https://blog.example.com/blog")browser_console(expression="window.location.href") — this reveals the real Framer slug, which is often different from what the display text suggests (e.g., display text: "Rust zero-cost abstractions vs. SIMD" → real slug: /blog/zero-cost)web_extract() with the discovered URL — it typically returns full content without needing the browser(function() {
const links = document.querySelectorAll('a');
const blogLinks = [];
links.forEach(a => {
if (a.href && a.href.includes('/blog/') && !a.href.endsWith('/blog')) {
blogLinks.push({text: a.textContent.trim().substring(0,80), href: a.href});
}
});
return JSON.stringify([...new Map(blogLinks.map(l => [l.href, l])).values()], null, 2);
})()
Fallback approaches (when browser is unavailable or too slow):
/sitemap.xml — Framer often auto-generates sitemaps with all post URLsweb_search to find cross-posts or summaries on other platforms (Substack, Medium, dev.to)Example from session (turbopuffer blog):
/blog/rust-zero-cost-abstractions-vs-simd → 404