wiki-anvil-researcher

name

wiki-anvil-researcher

description

Pre-spec research skill for wiki-anvil. Takes a GitHub repo URL, a tweet URL, or a local text-file path and builds up a durable research corpus under `research/` — per-target feature ledgers, an aggregated `FEATURES.md`, and a free-form `INSIGHTS.md`. The corpus is the input to a later interview-driven spec skill; this skill does NOT draft spec, plan, effort estimates, or roadmap. **Invoke explicitly** via the `/wiki-anvil-researcher` slash command or by saying "use the wiki-anvil-researcher skill on <url-or-path>" — auto-triggering on generic research prompts is unreliable because Claude can often do shallow research without a skill.

wiki-Anvil Researcher

Use this skill to turn a single input — a GitHub repository URL, a tweet URL, or a path to a local text file — into a durable, evidence-backed entry in wiki-anvil's research corpus. The corpus is the raw material that a later interview skill will use to shape the actual wiki-anvil spec. This skill's only job is to widen and deepen that corpus; it never decides scope, roadmap, or effort.

There is no feature spine — the corpus IS what we are building. Unlike spine-anchored researchers, we are enumerating observable behaviors across many comparable projects until patterns emerge. The operator is the one who reads FEATURES.md afterwards and decides what matters.

When to use

Use this skill when the user wants to:

• Pull a new reference project (repo, tweet, or text capture) into the wiki-anvil research corpus.
• Refresh an existing per-target ledger because the upstream has shipped new features.
• Capture a fleeting idea or discussion from Twitter / notes into INSIGHTS.md so it isn't lost.
• Populate the corpus ahead of the spec-interview phase.

Do NOT use this skill when:

• The user is debugging wiki-anvil itself — read the code, don't research sideways.
• The user wants a casual README summary of a repo — a plain Explore run suffices.
• The user is asking for spec drafting, effort estimates, bundle assignment, or roadmap synthesis — that is the job of a later interview skill that runs off the populated corpus, not this one.

Prerequisites

Before running the skill body, verify these preconditions. If any fails, stop and tell the user what to set up first — never fabricate an answer.

1. /home/oruc/Desktop/workspace/wiki-anvil/research/ exists. If it doesn't, create it — it is a plain directory inside the repo, not a submodule, and is safe to create on first run.
2. Required tools available: git, gh, defuddle (for tweet / page clean-markdown extraction), plus standard Unix utilities.
3. For repos with hundreds of source files, subagents must be available — the main context will not hold a full repo tree. If subagents are unavailable, fall back to colgrep-driven narrow reads and record the degradation in the run report.

Inputs

Exactly one per invocation:

• GitHub repo URL (e.g., https://github.com/breferrari/obsidian-mind) → full per-target extraction pass; writes research/<slug>.md and updates FEATURES.md + INDEX.md.
• Tweet URL (e.g., https://x.com/<user>/status/<id>) → defuddle extracts the tweet text; append to INSIGHTS.md.
• Local text-file path (e.g., /home/oruc/notes/random.txt) → read file; append to INSIGHTS.md as-is with a source line.

Optional:

• --refresh — rerun an existing <slug>; append a new row to ## Run history inside the per-target file rather than failing on collision.

The skill decides input type by sniffing the argument: github.com/<org>/<repo> → repo; hostname that looks like Twitter / X → tweet; an absolute filesystem path that exists and is readable → text file. Anything else is an error — ask the user to clarify rather than guess.

Outputs

Everything lives under /home/oruc/Desktop/workspace/wiki-anvil/research/:

1. <slug>.md — per-target ledger (GitHub input only). Created or updated.
2. FEATURES.md — aggregated, categorized corpus. Append-only by default. Every feature carries a seen-in: [<slug>, ...] back-reference list.
3. INSIGHTS.md — free-form notes from tweets and text captures. Append-only; never merged or deduplicated.
4. INDEX.md — flat chronological list of processed inputs (date, slug-or-source, kind). One row per run.
5. A short terminal report to the user (see §7 for the exact structured block).

Slug rules (GitHub targets only):

• <org>-<repo>, lowercased, hyphen-separated. Strip .git.
  • github.com/breferrari/obsidian-mind → breferrari-obsidian-mind.md
• On collision without --refresh, stop and ask — do not overwrite silently.
• Tweets and text files do not get a slug; they flow into INSIGHTS.md with an inline source line.

Confidence marker — the only classifier

There is no parity/stronger/weaker legend. Every feature entry carries exactly one confidence level:

• high — direct, unambiguous evidence. Code or docs explicitly describe the behavior.
• medium — evidence implies the feature but at least one observable behavior is uncertain.
• low — evidence is a README bullet, a marketing claim, or a forum post. Every low entry MUST surface in the per-target file's ## Needs follow-up section.

Confidence is orthogonal to category. A [capture] feature can be high or low; a [retrieval] feature can be high or low. The category tag is a filing hint, not a judgment.

Category tags — lightweight and optional

Each feature MAY carry zero or more of these tags, written inline next to the title. Pick what fits; don't force-fit if nothing applies.

Tag	When it fits
[capture]	Getting raw content (chat logs, clippings, voice notes) into the system.
[retrieval]	Surfacing stored content back — search, semantic lookup, context injection.
[ui]	Any user-facing surface — CLI, TUI, web, editor extension.
[ops]	Install, hooks, daemons, schedulers, cron, self-healing.
[infra]	Storage, indexing, databases, file-system layouts, sync.
[multi-agent]	Anything about multiple agents / LLMs collaborating or handing off.
[ingest]	Transforming raw captures into structured notes — parsers, extractors, frontmatter, routing.

Multiple tags are fine: a "Claude hook that writes frontmatter to a daily note" is reasonably [capture] [ops] [ingest]. Don't invent new tags in this pass — if something needs one, flag it in the run report and the operator decides later.

Evidence bar — non-negotiable

Every feature entry MUST carry all four evidence fields. If any field cannot be filled, drop confidence to low. Never silently ship a claim without evidence.

1. Source location — file path + line range, or URL + section anchor. Specific enough that a reader can verify in ≤30 seconds.
2. Direct quote — 1–3 line verbatim excerpt. Quote faithfully; do not paraphrase inside the blockquote.
3. Version / commit / fetch date — git short SHA + tag/branch + fetch date for repos; fetch date for tweets/pages.
4. Confidence — high / medium / low.

Coverage strategy — why enumeration needs machinery

No single LLM pass can guarantee full feature extraction from a non-trivial repo. The skill approaches coverage through five mechanisms. If any one is skipped, misses compound.

1. Two-pass design. Pass 1 is surface enumeration (walk the target's own source tree / docs / CLI systematically). Pass 2 is a target-first free walk — skim everything the target documents, with fresh eyes, and record anything Pass 1 didn't explicitly pick up. Pass 2 catches the "they do X and we haven't named X yet" items that enumeration misses.

2. Surface enumeration. Before feature extraction, enumerate every source surface that could reveal a feature, then hit each systematically. The full surface list lives in references/extraction-methods.md. At minimum, for repos: README headings, docs/ tree, CHANGELOG, release notes, CLI --help for every subcommand, HTTP/GraphQL route tables, OpenAPI/schema files, config schema (env vars + YAML + flags), test describe/it blocks, public API exports, top 20 production dependencies, issue labels, open-issue titles (≤100).

3. Parallel subagents — when you are the top-level agent. For non-trivial repos, Pass 1 spawns one subagent per surface cluster (routes, CLI+config, docs, tests, changelog+issues). Each returns a structured list with evidence. Main agent merges.

   Hard caveat: if you are yourself a subagent, do NOT fan out further. Nested subagent fan-out causes API Error: Stream idle timeout — partial response received on Sonnet-class subagents — the parent blocks waiting on children while its own API stream goes idle and gets killed (observed at ~39 min / 67 tool uses). When running as a subagent, do serial surface enumeration on yourself, cap features at 15–18 for coverage breadth, write incrementally (per-target file first with metadata block, then append features as you go), and respect a 10–12 min hard budget. A thin-but-shipped ledger beats a timed-out never-written one.

   Every subagent prompt MUST include the colgrep reminder block below, verbatim. Without it, subagents default to Grep/Glob and miss semantic matches. The same block lives in references/extraction-methods.md §"Subagent prompt template" as the canonical copy — either paste this one or the one there; they are kept identical. Do not paraphrase.

   This project has `colgrep` installed — a semantic code search tool.
   Use `colgrep` (via Bash) as your PRIMARY search tool instead of Grep/Glob.
   
   COLGREP COMMANDS:
   - Semantic search:      colgrep "error handling" -k 10
   - Regex + semantic:     colgrep -e "fn.*test" "unit tests"
   - Pattern only:         colgrep -e "async fn"
   - Search in path:       colgrep "query" ./src/api
   - Filter by type:       colgrep --include="*.rs" "query"
   - Multiple types:       colgrep --include="*.{ts,tsx}" "query"
   - List files only:      colgrep -l "query"
   - Exclude tests:        colgrep --exclude="*_test.go" "query"
   - Whole word:           colgrep -e "test" -w "testing"
   
   COLGREP BEHAVIOR:
   - First query may take 30-90 seconds (model loading + index building); subsequent queries are <5 seconds.
   - NEVER run colgrep in background mode — wait for the result.
   - NEVER fall back to Grep/Glob while colgrep is running.
   - If colgrep returns no results, try broader semantic terms or regex-only mode.
   
   DO NOT use Grep or Glob tools — use colgrep via Bash instead.
   

4. Coverage manifest as a first-class output. Every per-target file ends with ## Coverage manifest: a table of every surface scanned (with file count / status) and every surface skipped (with reason — "private", "404", "not applicable", "too-large-for-single-pass"). Under-coverage becomes visible and the operator can re-run with --refresh later.

5. Quality self-check gate. Before declaring done, the skill walks the checklist in §8. If any item fails, the skill reports what's incomplete rather than claiming success.

Treat these as hard requirements. If any mechanism is skipped or impossible (e.g., subagents unavailable), say so explicitly in the run report so the operator can calibrate trust.

Workflow

1) Detect input type and resolve the source

Branch on the argument:

• Matches github.com/<org>/<repo> → GitHub repo. Go to step 2a.
• Matches twitter.com/ or x.com/ → tweet. Go to step 2b.
• Absolute filesystem path that exists → text file. Go to step 2c.
• Anything else → stop, ask the user to clarify.

2a) GitHub repo — clone and pin

mkdir -p /tmp/wiki-anvil-research
DEFAULT_BRANCH=$(gh repo view <owner>/<repo> --json defaultBranchRef --jq '.defaultBranchRef.name')
git clone --depth 1 --branch "$DEFAULT_BRANCH" <repo-url> /tmp/wiki-anvil-research/<slug>
cd /tmp/wiki-anvil-research/<slug>
git rev-parse --short HEAD
git describe --tags --always

If the target is very large (>10k files) or the user explicitly asks for a lightweight pass, fall back to gh api — see references/extraction-methods.md §"For a GitHub repository".

Delegate crawl / tree-walk to an Explore subagent when the target is large — don't pollute the main context. Instruct the subagent to return a structured brief (surfaces found, doc pages, top-level code directories), not raw dumps. Tell it to use colgrep as its primary search tool. Continue to step 3.

2b) Tweet URL — defuddle-extract and append

# defuddle is NOT installed globally on this machine. Use the npx wrapper:
npx --yes defuddle-cli <tweet-url> > /tmp/wiki-anvil-research/<tweet-id>.md

X/Twitter caveat — know this before you run. X serves a JavaScript-rendered shell; static scrapers (including defuddle, curl, WebFetch) receive the HTML skeleton with wordCount: 0 and no tweet body. If defuddle returns an empty body:

1. Do NOT fabricate the tweet text. The skill's evidence bar forbids it; a reconstructed body from training knowledge is a silent integrity failure.
2. Record the failure verbatim in the INSIGHTS.md entry: source URL, fetch date, an explicit "Extraction status: FAILED — content could not be extracted" line, and a pointer to the fallback.
3. Fallback: if agent-browser is available, re-run with it — it drives a real browser session that sees rendered content. If not, ask the operator to paste the tweet text as a local text file and re-run as §2c.

Other tweet-adjacent surfaces that sometimes work without auth: nitter.* mirrors (flaky), vxtwitter.com / fxtwitter.com (redirect-and-render), Twitter's own cdn.syndication.twimg.com API. None are reliable; do not depend on them.

Read the extracted markdown. Write an entry to INSIGHTS.md:

## <YYYY-MM-DD> — tweet by @<handle>

- **Source:** <tweet-url>
- **Fetched:** <YYYY-MM-DD>

> <verbatim tweet text, 1–20 lines>

<1–4 free-form sentences: what caught your attention; connections to other corpus entries if obvious; no synthesis, no ranking.>

Update INDEX.md with a row (<YYYY-MM-DD> · tweet · @<handle>/<tweet-id> · <one-line topic>). Skip to §7 (report back).

2c) Local text-file path — read and append

Read the file verbatim. Write an entry to INSIGHTS.md:

## <YYYY-MM-DD> — text capture: <filename>

- **Source path:** <absolute path>
- **Captured:** <YYYY-MM-DD>

<verbatim file content — wrap long lines in a blockquote or fenced block if it improves readability, but do not paraphrase or summarize>

<0–3 optional free-form sentences of immediate reaction; skip if the content speaks for itself.>

Update INDEX.md with a row (<YYYY-MM-DD> · text · <filename> · <one-line topic>). Skip to §7.

3) Enumerate surfaces (GitHub path only)

Work through the surface list in references/extraction-methods.md. For each surface, decide:

• In scope — scan and record findings.
• Not applicable — mark as skipped with reason in the coverage manifest (e.g., "no CLI — pure library").
• Blocked — paywalled, 404, rate-limited. Mark with reason.

For non-trivial targets, fan out the surface enumeration to subagents in parallel — one per surface cluster. Paste the colgrep reminder block verbatim into every subagent prompt.

4) Extract observable-behavior features (GitHub path only)

For each surface, enumerate features with the template in references/template-target.md. Each feature has:

• Title (short, scannable).
• Category tags (0+, optional).
• Confidence marker (high/medium/low).
• Description (2–3 sentences — what this feature actually does).
• How it works (2–4 sentences — mechanism, key files, data flow).
• Observable behaviors (minimum 2, aim for 4–7 — concrete rules, limits, defaults, UX specifics). If you can't list at least 2, the entry is under-researched — go dig.
• Evidence block (all four fields).
• Searched for (what queries you ran; helpful for low-confidence entries).

5) Write the per-target file (GitHub path only)

Use references/template-target.md verbatim as the skeleton. Fill every section. Key rules:

• One heading per feature.
• Observable behaviors list is not optional.
• Evidence must be specific (file:line or URL#anchor).
• Coverage manifest at the bottom lists every surface attempted and every surface skipped with reason.
• Run history table at the bottom — new row on --refresh, never overwrite prior rows.

6) Update FEATURES.md (GitHub path only)

FEATURES.md is the critical aggregate artifact. After writing the per-target file, walk its features one by one and update FEATURES.md:

1. For each new feature in the per-target file:

   • Search FEATURES.md for a likely semantic match (same category tags, similar observable behaviors, near-identical title).
   • If no plausible match exists → append a new row. Seed seen-in: [<slug>].
   • If an exact-or-obvious match exists → add the current slug to its seen-in: list. Leave the canonical description alone.
   • If a match is semantically similar but not obviously the same → append as a new row (default is "don't merge silently") AND list the pair in the run report's Possible merges: block so the operator can confirm or reject.

2. Never silently merge. The rule exists because early in the corpus, what looks like duplication often turns out to be a real behavioral split once more targets arrive.

3. Never rewrite FEATURES.md wholesale. Parse it, update surgically, write back. Destructive rewrites erase seen-in: lists that other runs depended on.

See references/template-features.md for the exact row format.

7) Update INDEX.md

Append one row to INDEX.md per run. The index is a flat chronological list, not a matrix. Columns: date · kind (repo / tweet / text) · slug-or-source · one-line topic or commit SHA.

Do not rewrite INDEX.md from scratch. On --refresh, still append a new row — every run is a row.

8) Report back to the user

Produce this structured block — the operator reads it at a glance, and it diffs cleanly across runs. Keep the shape, fill every field. If a field has nothing to say, write none rather than omitting the line.

Delivery: text first, file second. The canonical delivery of the run report is the text you return to the caller (printed in your final response, or returned as the subagent summary if this skill is running inside one). Additionally, best-effort save the same block as research/report.md (or whatever output directory is in effect) so it's reviewable later. Some harness contexts block subagents from writing files — if file-write fails or is refused, do NOT treat that as an error; the text delivery is sufficient on its own. Just note (report delivered inline — file-save unavailable) in your final message so the operator knows not to look for the file.

**Input:** <repo-url | tweet-url | text-path> (`github-repo | tweet | text`)
**Target slug / source:** `<slug or filename>` · <version @ sha | fetched YYYY-MM-DD>
**Files written:**
- `research/<slug>.md` (created | updated, run N) — GitHub only
- `research/FEATURES.md` (+<n> new rows, <n> seen-in updates) — GitHub only
- `research/INSIGHTS.md` (+1 entry) — tweet / text only
- `research/INDEX.md` (+1 row, always)
**Features extracted (GitHub only):** total <n> · high <n> · medium <n> · low <n>
**Category mix:** [capture] <n> · [retrieval] <n> · [ui] <n> · [ops] <n> · [infra] <n> · [multi-agent] <n> · [ingest] <n> · untagged <n>
**Coverage:** <n>/<n> surfaces scanned · skipped: <short list or "none">
**Possible merges (for operator review):**
- "<new feature title>" looks semantically close to existing "<existing title>" in FEATURES.md — appended as new; confirm if you want to merge
- … (0 or more)
**Blockers / gaps:** <e.g., "gh rate-limited on open issues", "docs/compliance behind login", or "none">

For tweet / text inputs, skip the "Features extracted", "Category mix", "Coverage", and "Possible merges" lines — write n/a (insights-only capture) on a single line instead. Keep the rest.

Quality self-check before declaring done

Before claiming success, verify:

• Correct input branch was taken — a tweet URL did NOT trigger a per-target extraction.
• For GitHub inputs: research/<slug>.md exists and parses as valid Markdown.
• Every feature entry has all four evidence fields populated (source, quote, version/date, confidence).
• Every feature entry has ≥2 observable behaviors. Entries with <2 are under-researched — re-dig.
• ## Coverage manifest is populated with every surface attempted AND every surface skipped-with-reason.
• For GitHub inputs: FEATURES.md was updated in place — new features appended, existing matches had seen-in: extended. Not a destructive rewrite.
• FEATURES.md row counts reconcile: seen-in: for this slug appears on ≥1 row (otherwise this run contributed nothing, which is structurally suspicious for a non-trivial repo).
• Possible-merges section in the run report lists every case where the skill appended a row instead of merging because semantic similarity was uncertain.
• For tweet / text inputs: INSIGHTS.md entry contains the verbatim source material plus a dated source line. No paraphrasing inside the blockquote.
• INDEX.md got exactly one new row — flat list, appended, not rewritten.
• Run history row appended to the per-target file (GitHub only) on refresh runs.
• The report in §7 was printed — totals match the actual file contents.
• Zero spec drafting, zero roadmap, zero effort estimates have been written anywhere. If the skill started to draft any of these, it stopped and noted the scope boundary.

Treat this checklist as a hard gate. If any box can't be ticked, say so in the report instead of declaring success.

Reference files

For the exact shape of the per-target file, FEATURES.md rows, and the extraction methodology, see:

• references/template-target.md — per-target ledger skeleton, observable-behaviors format, coverage manifest, run history.
• references/template-features.md — FEATURES.md row format, seen-in: back-reference rules, append-vs-merge policy.
• references/extraction-methods.md — surface enumeration, subagent delegation pattern, the colgrep reminder block (paste verbatim into every subagent prompt).