agent-wiki-ingest

name	agent-wiki-ingest
description	Ingest one or more agent trajectories (raw bob/claude traces or normalized JSON) into an agent-wiki end-to-end — convert, summarize, extract guidelines, synthesize skills, consolidate into clusters, and catalog. Use when you have a batch of traces to turn into a wiki in one pass.

Agent Wiki — Ingest (end-to-end orchestrator)

Overview

This is the one-pass entry point for turning a batch of raw trajectories into a fully-built wiki. It orchestrates the rest of the agent-wiki family in the right order so no pass is skipped — in particular the cross-trajectory consolidation pass, which is easy to forget when each skill is invoked by hand.

You — the driving agent — run this by spawning one subagent per (trace × pass), not by doing the work inline. That keeps your own context small (you never load every trace's full JSON) and lets independent passes run in parallel. Each subagent acts as the corresponding single-purpose skill (agent-wiki-summarize, -extract-guidelines, -synthesize-skill, -consolidate-guidelines); this skill only sequences them and passes the per-trace adapter notes.

The pipeline:

0.  Convert    raw bob / claude traces → normalized analysis JSON   (skip if already normalized)
1.  Bootstrap  create wiki scaffold + seed catalog                  (skip if wiki exists)
1.5 Skip       drop traces whose summaries/<sid>.md already exists   [pre-flight — idempotency]
2.  Summarize  1 subagent / new-trace → summaries/<sid>.md          [PARALLEL]
3.  Extract    1 subagent / new-trace → guidelines/*.md (+tags)     [SEQUENTIAL]
4.  Synthesize 1 subagent / new-trace → skills/<slug>/ --archive-covered  [SEQUENTIAL]
5.  Consolidate 1 subagent over the whole corpus → cluster pages    [SINGLE — MANDATORY]
6.  Catalog    final bookkeeping → indexes, used-by, priority       [you run this directly]

Idempotent by default. Re-running on the same source dir reprocesses nothing: Step 1.5 filters out every trace that already has a summary page, so Steps 2–4 only touch genuinely new traces. The consolidate + catalog tail always runs (it's cheap and self-idempotent). To force a redo of an already- ingested trace, keep it in the list and pass --rewrite to its render-* calls.

Why this order. synthesize-skill runs before consolidate-guidelines so skills claim recipe-level territory first (and archive the atomics they cover via --archive-covered); consolidation then clusters only the surviving atomics. This matches the consolidate skill's own rule — "don't propose clusters that overlap a skill's territory."

Why parallel vs sequential. Summarize writes one independent file per trace (summaries/<sid>.md) → safe to parallelize. Extract and synthesize both mutate shared state (guidelines/_id_index.json, skills/_id_index.json, _config.yaml, and the _archived/ moves) → run them one trace at a time to avoid lost-update races.

Input

One of:

• a list of trace file paths
• a directory of traces (the skill globs it)
• already-normalized analysis JSON files

…plus a target --wiki-root (e.g. wiki-twobatch-skills).

Detecting trace shape (Step 0 dispatch)

Read the top-level JSON keys of each input to classify it:

Shape	Signature	Conversion
bob session JSON	top-level sessionId + messages	bob-trace-converter
claude stream-json	JSONL lines with {"type":"system"/"assistant"/"result"}	normalize_stream_json_transcripts.py
normalized analysis JSON	top-level model + messages + metadata.id	pass through (no conversion)

Step 0 — Convert

Write converted output under a stable corpus dir: trajectories/normalized/<label>/items/.

bob session JSON:

NODE_OPTIONS='' node ~/.claude/skills/bob-trace-converter/scripts/convert_bob_trace.mjs \
  <trace.json> --out-dir trajectories/normalized/<label>/items --format both

The NODE_OPTIONS='' prefix is required — some shells inject a --require preload that breaks a bare node invocation. Strip it for this call.

The converter writes three files per trace; the ingest pipeline consumes the *-openai-chat-completions.analysis.json one.

claude stream-json:

uv run python explorations/agent-wiki/experiments/harness/normalize_stream_json_transcripts.py \
  --in <transcripts-dir> --out trajectories/normalized \
  --label <label> --user-prompt "<the task prompt>"

Already normalized: skip — use the path as-is.

Collect the resulting list of analysis-JSON paths; this is the trace set the rest of the pipeline iterates.

Step 1 — Bootstrap the wiki

If <wiki-root>/_index.jsonl does not exist:

mkdir -p <wiki-root>/{summaries,guidelines,tasks,skills}
uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py \
  --wiki-root <wiki-root> catalog

The first catalog seeds AGENTS.md and _config.yaml from the bundled defaults and writes empty indexes. Skip this whole step if the wiki already exists — you're appending to it.

Piping JSON to the helper — avoid echo

Every render-* subcommand reads JSON on stdin. The echo '<json>' | … form in the per-pass skills breaks when the payload has multi-line content/narrative fields (literal newlines become invalid control characters in the shell-quoted string). Tell every subagent to write its payload to a temp file and cat it instead:

cat /tmp/ingest-payload.json | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-guidelines

Step 1.5 — Skip already-processed traces (pre-flight)

This is what makes re-running the skill on the same source dir cheap. The helper's render-* subcommands skip-if-exists, but only after a subagent has already read the trace and synthesized its output — so the LLM cost is already spent. Filter before spawning any subagent.

For each normalized trace, read its session_id — it lives at metadata.id (bob-converted analysis JSON) or top-level session_id (claude-normalized). If <wiki-root>/summaries/<sid>.md already exists, the trace was ingested on a prior run → drop it from the work-list. The surviving new-trace list is what Steps 2–4 iterate.

Compute the new-trace list and log what was skipped (never let a silent no-op masquerade as success):

for f in <trace-glob>; do
  sid=$(uv run python -c "import json,sys; d=json.load(open(sys.argv[1])); print(d.get('session_id') or d.get('metadata',{}).get('id',''))" "$f")
  if [ -n "$sid" ] && [ -f "<wiki-root>/summaries/$sid.md" ]; then
    echo "skip (already ingested): $sid  $f"
  else
    echo "NEW: $sid  $f"
  fi
done

The NEW: lines are the work-list for Steps 2/3/4. If every trace is skipped, that's fine — jump straight to Steps 5–6 (the tail always runs).

Override. To force reprocessing of an already-ingested trace, keep it in the work-list and pass --rewrite to its render-* calls (the helper overwrites instead of skipping).

Step 2 — Summarize (parallel subagents)

Spawn one subagent per new-trace (from Step 1.5's work-list), all in parallel. Each acts as agent-wiki-summarize (point it at that skill's SKILL.md). In each subagent prompt include:

• the analysis-JSON path and the --wiki-root
• the trace's agent (bob, claude-code, …) — it must set agent: accordingly, not hardcode claude-code
• the bob field-mapping adapter notes (only if the trace came from bob): session_id ← metadata.id; model ← top-level model; tool calls live in messages[i].content[j] blocks with type: "tool_use"; transcript_path ← metadata.source_file; recalled_guidelines is empty for a freshly-built wiki
• do NOT run catalog — the orchestrator runs it once at the end

Each subagent pipes its summary JSON to:

echo '<json>' | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-summary

Step 3 — Extract guidelines (sequential subagents)

Spawn one subagent per new-trace, one at a time (wait for each before starting the next — they share guidelines/_id_index.json and _config.yaml). Each acts as agent-wiki-extract-guidelines. In each prompt:

• the analysis-JSON path, --wiki-root, agent, and bob adapter notes
• the list of existing guideline slugs (from prior traces this run) so it suppresses near-duplicates
• instruct it to attach a tags: array to every entity (these now propagate to both the .md frontmatter and _config.yaml — see commit that fixed render-guidelines)
• instruct it to set "agent": "<source>" on every entity. The extract-guidelines entity schema does not list agent as a field, so the subagent must add it explicitly; otherwise the page defaults to agent: claude-code even for bob traces.
• skip arc: for single-summary sessions
• do NOT run catalog

Pipe via:

echo '<json>' | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-guidelines

Step 4 — Synthesize skills (sequential subagents)

Spawn one subagent per new-trace, one at a time (shared skills/_id_index.json plus _archived/ moves). Each acts as agent-wiki-synthesize-skill. In each prompt:

• the analysis-JSON path, --wiki-root, agent, bob adapter notes
• the list of existing skill slugs so it doesn't re-author one
• tell it to decide promote-vs-skip per that skill's "When To Use" rubric (trivial single-command recipes → skip and emit nothing)
• when promoting, pipe with --archive-covered so the atomics the skill subsumes are soft-archived:

  cat /tmp/skill-payload.json | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-skill --archive-covered
  

  --archive-covered is safe to run blind: the matcher only archives an atomic from another trajectory when the skill's tags are a true superset (≥2 non-generic shared tags). The weak lexical heuristics (a slug word or format token appearing in the atomic's title) fire only for atomics from the same trajectory this skill was synthesized from, so a skill can no longer reach across into an unrelated trace's atomic on a coincidental word like "python" or "csv".
• do NOT run catalog

Step 5 — Consolidate (single subagent — MANDATORY)

This pass is not optional. It is the step most easily forgotten when the family is invoked by hand, and it is the whole reason this orchestrator exists. Always run it, even on a small corpus — the subagent's own judgment returns zero clusters when nothing qualifies, which is the correct outcome for a tiny or heterogeneous corpus.

Run it even when Step 1.5 skipped every trace. A re-run that ingests no new traces still benefits from a consolidation pass over the existing corpus — it can form clusters that an earlier run missed. Steps 5 and 6 are the always-on tail; only Steps 2–4 are gated on the new-trace list.

Spawn one subagent acting as agent-wiki-consolidate-guidelines over the whole surviving-atomic corpus. In its prompt:

• the --wiki-root
• instruct it to run dump-guidelines first, then propose clusters
• remind it: a cluster needs ≥2 atomic members sharing a real rule (not just a topic); don't propose clusters overlapping a skill's territory (the skill is already the canonical aggregator)
• do NOT run catalog — the orchestrator runs it next

Each cluster is piped via:

echo '<json>' | uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> render-cluster

Step 6 — Catalog (you run this directly)

One final bookkeeping pass:

uv run python explorations/agent-wiki/skills/scripts/build_agent_wiki.py --wiki-root <wiki-root> catalog

This regenerates _index.jsonl, the section indexes, the priority table, the "By tag" and used-by sections, and propagates cluster: / superseded_by: backrefs onto clustered atomics.

Report

After Step 6, report the end-state counts: summaries, surviving atomics, clusters, skills, archived atomics — and call out any trace that produced no guidelines or no skill (trivial recipes), plus any cluster proposals that were considered and rejected.

Best practices

1. Consolidation is mandatory. Step 5 always runs. The cluster subagent self-skips individual clusters; the pass never skips.
2. One subagent per (trace × pass). Don't batch multiple traces into one subagent — it bloats context and muddies provenance.
3. Parallel only for summarize. Extract, synthesize, and consolidate all touch shared index/config state — keep them sequential.
4. Subagents never catalog. Only the orchestrator does, once, at the end. A mid-run catalog wastes work and can race with in-flight writes.
5. Pass agent: through. Bob traces are bob, not claude-code. The summarize and extract subagents must stamp the right source.
6. Tags on every guideline. They drive the "By tag" index and future cluster formation; an untagged atomic is invisible to tag-based recall.
7. Idempotent by default. Step 1.5 skips any trace that already has a summaries/<sid>.md, so re-running on the same source dir reprocesses nothing. Use --rewrite on the render-* calls to force a redo of a specific trace.