wiki-research-loop

# Wiki Research Loop Driver that turns a wiki into an auto-grown knowledge base. Layers on top of `wiki-builder` and `wiki-query`. ## Loop semantics ``` seed-queue (pending) → next-seed → fetch sources via plugins (web | arxiv | github) → extract claims → dedupe vs index (FTS5; later vector via 3.3.2) → compile new page or amend existing → upsert page (auto-FTS-index) → enqueue follow-up seeds (max-depth gate) → mark seed done → if budget OR convergence OR kill-switch → halt ``` ## Halt conditions (any one trips) - `budget_usd` exceeded (loop tracks per-fetcher cost estimate) - `max_pages_per_run` written - `max_depth` reached on every active branch - 3 consecutive pages add < 5 % new claims (convergence) - File `~/.pro-workflow/STOP` exists (operator kill-switch) - `wiki.config.md` `auto_research.enabled: false` - Wiki `private: true` AND any non-local fetcher selected ## Commands In a plugin session, call the `providers` MCP server's `run_provider_task` tool with `task: "research"` and the runner arguments below, for example `args: ["run", "agent-memory", "--fetchers", "github"]`. Its optional GitHub token comes from the plugin configuration dialog. The direct CLI uses `PRO_WORKFLOW_GITHUB_TOKEN` when explicitly supplied. Never retrieve tokens from existing machine credentials. ``` node $SKILL_ROOT/scripts/research-loop.js run <slug> [--max-pages N] [--max-depth N] [--budget-usd 0.50] [--fetchers web,arxiv,github] node $SKILL_ROOT/scripts/research-loop.js seed <slug> "<query>" [--depth 0] [--parent-id N] node $SKILL_ROOT/scripts/research-loop.js seeds <slug> [--status pending|active|done|failed] node $SKILL_ROOT/scripts/research-loop.js cancel <slug> node $SKILL_ROOT/scripts/research-loop.js status ``` CLI flags override `wiki.config.md` for one run only. ## Source fetchers Pluggable. Each lives at `scripts/source-fetchers/<name>.js`. Interface: ```js module.exports = { name: 'web', match: (q) => true, // is this fetcher useful? estimateCost: (q) => ({ usd: 0, tokens: 0 }), fetch: async (q, opts) => [ // returns RawDoc[] { url, title, content, fetched_at } ] }; ``` Built-in: - **`web.js`** — Fetches via the user's available `WebFetch` tool through a stdin/stdout shim. Treats result as plain text/markdown. - **`arxiv.js`** — `https://export.arxiv.org/api/query` (free, public, no key). Returns abstract + metadata. - **`github.js`** — `https://api.github.com/search/repositories` + README pull (uses the explicitly configured GitHub token, otherwise unauthenticated rate limit). Standalone CLI runs load custom fetchers from `~/.pro-workflow/fetchers/<name>.js`. The plugin MCP server uses only bundled fetchers so configured tokens are not handed to user-supplied modules. ## Budget enforcement Pre-iteration: sum `estimateCost` across selected fetchers. If projected cumulative cost would exceed `budget_usd`, halt. Post-iteration: track tokens used by the LLM compile step (Anthropic/OpenAI passthrough). Hard-kill on overrun. Per-fetcher overrides via env: `WIKI_LOOP_BUDGET_USD`, `WIKI_LOOP_MAX_PAGES`, `WIKI_LOOP_MAX_DEPTH`. ## Seed queue SQLite-backed via `wiki_seeds` table: | field | meaning | |-------|---------| | `query` | natural-language seed | | `status` | `pending` → `active` → `done`\|`failed` | | `parent_id` | seed that produced this one | | `depth` | BFS depth from root | Loop pops by `(depth ASC, created_at ASC)` so it explores breadth-first. ## Convergence detection After each compiled page, compute Jaccard overlap of claim-text tokens vs the prior 3 pages. If `< 5 %` novel content for 3 consecutive pages, halt and report `converged`. ## Kill switch ``` touch ~/.pro-workflow/STOP ``` Loop checks per-iteration and halts gracefully. Remove file to resume next run. ## Privacy guard If `wiki.config.md` has `private: true`, the loop refuses any non-local fetcher and emits a warning. Only `raw/` ingestion via manual seeds is allowed. ## Reactive trigger (Phase 3.3.4) `scripts/file-watcher.js` watches `wiki/<slug>/wiki/**/*.md`. On user-edited claim, enqueues a verification seed (`verify: <claim>`) at depth 0. Wired through pro-workflow's `file-watcher.js` hook. ## Cron tick (Phase 3.3.4) `scripts/research-tick.js` is launchable from any cron-style runner. Picks the oldest opted-in wiki with pending seeds and runs a single iteration. Hook event: `pro-workflow:research-tick`. ## Output Each run writes: ``` <wiki-root>/logs/research-<UTC-timestamp>.md # human-readable run log <wiki-root>/derived/run-<UTC-timestamp>.json # structured stats ``` Run log lines: ``` [2026-05-08T10:42Z] seed-3 (depth=1) "memory consolidation in agents" fetcher=arxiv hits=3 fetcher=web hits=2 compiled wiki/concepts/memory-consolidation.md (claims=7, novel=4) enqueued 2 follow-up seeds cost so far: $0.04 / $0.50 ``` ## Integration with `wiki-query` Every compiled page goes through `wiki-cli.js page` so FTS5 stays consistent. The dedupe step calls `searchWiki` with the candidate claim text to find near-duplicates. ## Status (Phase 3.3.1) Ships: loop driver, seed queue, web/arxiv/github fetchers, budget caps, convergence detector, kill-switch, manual `run` command. Defers: - Vector dedupe (Phase 3.3.2 via sqlite-vec) - LLM-judged claim novelty (current = Jaccard token overlap) - Cron + reactive (Phase 3.3.4)