knowledge-distill

name	knowledge-distill
description	Phase 4.5 of the inverted pipeline (between ingest and compose). Distills the run's source claims into recurring type:concept / type:entity / type:person pages, creating-or-merging them under a lock with claim-level dedup so the bound wiki compounds across runs (distilled pages get enriched, not duplicated). Also synthesizes a citable answer_claims: surface onto each type:question node from its findings' claims (Step 6.1). An optional cross-lingual pass merges DE↔EN twin claims on mixed-language bases (auto-skips otherwise). Fail-soft and optional: a distill failure never blocks compose. Use this skill whenever the user says 'distill the concepts', 'build the concept web', 'phase 4.5', 'knowledge distill', 'extract entities and concepts', 'answer the question nodes', or 'dedupe claims'. After distill, knowledge-compose reads the distilled pages as framing context.
allowed-tools	Read, Write, Bash, Task

Knowledge Distill

Phase 4.5 of the inverted pipeline — it sits between knowledge-ingest (Phase 4) and knowledge-compose (Phase 5): plan → curate → fetch → ingest → **distill** → compose → verify → finalize.

Phase 4 deposits one type: source page per fetched URL (verbatim body + pre_extracted_claims:). That makes the wiki a citation store. This phase turns those source claims into the distilled concept/entity web that makes a Karpathy wiki compound across runs: type: concept / type: entity pages that successive runs enrich (new claims appended, source backlinks unioned) rather than duplicate, with claim-level dedup at deposit and per-run re-narration of updated summaries (Step 6.7) so the wiki compounds narratively — the entry-point prose integrates new evidence — as well as structurally. The differentiation thesis's compounding loop + claims-dedup metric become real here.

Optional + fail-soft. A distill failure must NEVER block knowledge-compose. Same posture as the knowledge-curate Step 0.5 coverage pre-step: warn loudly, exit cleanly, let the pipeline continue. The concept/entity pages are an enrichment layer, not a correctness gate.

Read ${CLAUDE_PLUGIN_ROOT}/references/inverted-pipeline.md §"Phase 4.5 — knowledge-distill" once to anchor on the contract.

When to run

• <project>/.metadata/ingest-manifest.json exists with non-empty ingested[] (Phase 4 has run).
• User explicitly invokes /cogni-knowledge:knowledge-distill, or the pipeline reaches Phase 4.5.

Never run when

• No <project>/.metadata/ingest-manifest.json or ingested[] empty — offer knowledge-ingest first (but do NOT hard-fail the pipeline; distill is optional).
• No binding.json at the resolved knowledge root — offer knowledge-setup first.
• binding.wiki_path does not resolve to a directory containing .cogni-wiki/config.json — the binding is stale.

Parameters

Parameter	Required	Description
--knowledge-slug	Yes	Slug of the bound knowledge base.
--project-path	Yes	Absolute path to the project directory.
--knowledge-root	No	Override the default knowledge-base directory.
--no-renarrate	No	Skip Step 6.7 (summary re-narration). Default is on — narrative compounding is the point of Phase 4.5; this opt-out exists for byte-stable re-runs / cost control.
--no-crosslingual	No	Skip Step 6.6 (cross-lingual DE↔EN claim merge). Default is on but auto-skips with zero LLM cost when no candidate pairs exist (every single-language base); this opt-out forces it off even on a mixed-language base.
--dry-run	No	Print the resolved inputs (bundle source count, existing distilled-page count, bundle hash, resume verdict) without dispatching the distiller.

Workflow

0. Pre-flight

Required plugins. Probe only cogni-wiki (clean-break), byte-for-byte the probe knowledge-ingest uses:

probe_plugin() {
  local plugin="$1" skill="$2"
  test -f "${CLAUDE_PLUGIN_ROOT}/../${plugin}/skills/${skill}/SKILL.md" && return 0
  for d in "${CLAUDE_PLUGIN_ROOT}/../../${plugin}/"*/skills/"${skill}"/SKILL.md; do
    [ -f "$d" ] && return 0
  done
  return 1
}
probe_plugin cogni-wiki wiki-ingest && WIKI_OK=yes || WIKI_OK=no

If WIKI_OK=no, warn and exit cleanly (distill is optional — do not block the pipeline).

Resolve the cogni-wiki script dir. Use the SAME resolve_wiki_scripts helper knowledge-ingest / knowledge-finalize use (shared byte-for-byte) — it picks the newest numeric version dir. concept-store.py merge needs this dir for _wiki_lock / is_foundation_page / parse_frontmatter, and Step 6 needs backlink_audit.py / wiki_index_update.py / config_bump.py:

. "${CLAUDE_PLUGIN_ROOT}/scripts/resolve-wiki-scripts.sh"
WIKI_INGEST_SCRIPTS=$(resolve_wiki_scripts wiki-ingest backlink_audit.py) || { echo "⚠ cogni-wiki wiki-ingest scripts not found — skipping distill (optional)"; exit 0; }

Binding + wiki root. Resolve knowledge_root (same logic as knowledge-ingest). Read the binding via knowledge-binding.py read --knowledge-root <knowledge_root>. On success: false → warn + exit cleanly. Parse data.binding.wiki_path as WIKI_ROOT; confirm <WIKI_ROOT>/.cogni-wiki/config.json and <WIKI_ROOT>/wiki/ exist and are writeable.

Project manifests. Read <project_path>/.metadata/ingest-manifest.json. If absent or ingested[] is empty → warn ("nothing to distill — run knowledge-ingest first") and exit cleanly. Read <project_path>/.metadata/plan.json for TOPIC, output_language (default en), and derive the project slug (the project directory's basename, matching what knowledge-finalize uses for derived_from_research:).

1. Build the claim bundle (the distiller's only evidence)

For each entry in ingested[], read its wiki/sources/<slug>.md page and pull its pre_extracted_claims: via the shared parser. Write a compact, bounded bundle file so the distiller reads ONE file instead of N source pages. Pass paths via env vars so spaces/apostrophes can't break the literal:

Run the claim bundle subprocess in references/distill-bundle-builders.md §1 — env inputs and the verbatim code are there; it prints the count the next paragraph captures.

Capture the printed source count. If it is 0 (no source carries claims) → warn ("no source claims to distill") and exit cleanly.

Compute a content hash of the bundle for the resume check (Step 3). It MUST hash the bundle's bytes, not a path — fetch-cache.py key hashes a URL string and would make the resume check path-keyed (a changed claim set on the same path would falsely look "unchanged"), so do not use it here:

SHA=$(python3 -c 'import hashlib,sys;print(hashlib.sha256(open(sys.argv[1],"rb").read()).hexdigest())' "<project_path>/.metadata/distill-bundle.txt")

2. Build the existing concept/entity slug index

So the distiller can reuse an existing concept's title (landing the merge on the same page = compounding) and avoid proposing a near-duplicate:

Run the existing concept/entity slug index subprocess in references/distill-bundle-builders.md §2 — env inputs and the verbatim code are there; it prints the count the next paragraph captures.

3. Resume check (idempotent no-op)

If <project_path>/.metadata/distill-manifest.json exists AND it records the same bundle_hash as SHA from Step 1 → the distiller already ran on this exact claim set. Skip re-dispatch (the LLM is the expensive part) and jump to the final summary, reporting "no-op (bundle unchanged)". concept-store.py merge is itself byte-stable on re-run, so skipping is purely a cost optimization, not a correctness requirement.

(The orchestrator stores SHA by reading it back from the manifest's bundle_hash field — Step 6 threads it in. On the first run the manifest has no bundle_hash, so the check fails and the distiller runs.)

If --dry-run: print the source count, existing distilled-page count, SHA, and the resume verdict, then stop.

4. Initialize distill-manifest.json

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/concept-store.py init --project-path <project_path>

4.5. Build the answer-bundle (prerequisite for the parallel dispatch)

The answer-distiller (dispatched in the Step 5 fan-out wave) gives each type: question node (deposited by knowledge-ingest Step 4.5) a citable answer surface — an answer_claims: frontmatter block distilled from the node's findings' pre_extracted_claims:, exactly as concept/entity pages got distilled_claims:. A question node becomes a first-class cross-source answer unit the composer can later cite and the verifier can score. Its bundle reads only wiki/questions/*.md + wiki/sources/*.md — disjoint from the concept-distiller's claim bundle and dependent on no distiller output — so it is built here, before the fan-out, and the answer-distiller rides the same wave as the concept-distiller. This step needs no index/backlink/entries_count work — question nodes already exist and are indexed at ingest time; it only prepares their answer-claim enrichment.

Skip cleanly when <WIKI_ROOT>/wiki/questions/ does not exist or is empty (a base that predates the question-node feature, or a run with no question nodes) — set ANSWER_Q_COUNT=0 (the Step 5 wave then dispatches only the concept-distiller) and continue.

Build the per-question claim bundle. For each wiki/questions/<slug>.md, read its sources_answering: list and pull each listed source page's pre_extracted_claims:, emitting one ## question: <slug> | <title> block followed by the same 3-part <source_slug> | <claim_id> | <text> claim lines the Step-1 concept bundle uses. Reuse the shared parsers — split_frontmatter (the same _wikilib import question-store.py emit uses, for the inline sources_answering: list) + parse_pre_extracted_claims:

Run the answer bundle subprocess in references/distill-bundle-builders.md §5 — env inputs and the verbatim code are there; it prints the count the next paragraph captures.

Capture the printed count as ANSWER_Q_COUNT. If it is 0 (no question node carries answerable claims) → keep ANSWER_Q_COUNT=0; the Step 5 wave dispatches only the concept-distiller and Step 6.1 reports n/a (no answerable question nodes).

5. Fan-out wave — dispatch concept-distiller + answer-distiller in parallel

The concept-distiller and the answer-distiller read disjoint inputs (the source-claim bundle vs the answer-bundle, both built from already-ingested pages) and write disjoint targets (distill-records.txt → wiki/{concepts,entities,people}/ vs answer-records.txt → wiki/questions/ answer_claims:); neither consumes the other's output, and neither LLM Task acquires the wiki lock (only the downstream merge scripts do). So dispatch both in one fan-out wave — a single assistant message with two Task calls — instead of back-to-back, mirroring the curate/ingest fan-out posture. Gate the answer-distiller leg on ANSWER_Q_COUNT > 0 from Step 4.5 (when it is 0, the wave carries only the concept-distiller):

Task(concept-distiller,
     CLAIM_BUNDLE_PATH=<project_path>/.metadata/distill-bundle.txt,
     SLUG_INDEX_PATH=<project_path>/.metadata/distill-slug-index.txt,
     RECORDS_OUTPUT_PATH=<project_path>/.metadata/distill-records.txt,
     OUTPUT_LANGUAGE=<plan.json::output_language, default en>)
Task(answer-distiller,                                   # only when ANSWER_Q_COUNT > 0
     ANSWER_BUNDLE_PATH=<project_path>/.metadata/answer-bundle.txt,
     RECORDS_OUTPUT_PATH=<project_path>/.metadata/answer-records.txt,
     OUTPUT_LANGUAGE=<plan.json::output_language, default en>)

Both agents live under ${CLAUDE_PLUGIN_ROOT}/agents/ (concept-distiller.md / answer-distiller.md) — dispatched via Task, not Skill. Each is a single pass that reads its bundle and writes its raw-text records file. After both return, parse each envelope independently:

• concept-distiller — ok: true → its records feed Step 6; ok: true, concepts_proposed: 0 → nothing to distill (skip Step 6's merge); ok: false → warn (surface the error) and treat the concept path as empty (distill is optional).
• answer-distiller (dispatched only when ANSWER_Q_COUNT > 0) — parse like the concept leg so a legitimate no-op is not surfaced as an error: ok: true → its records feed Step 6.1; ok: true, questions_proposed: 0 → benign n/a skip (no question had an answerable claim), Step 6.1 reports n/a; ok: false → warn (surface the error) and skip Step 6.1's answer-merge. Answer synthesis never blocks the pipeline.

If the concept leg proposed nothing and the answer leg was skipped/empty/ok: false, jump straight to the final summary. Otherwise continue to Step 6 (concept merge) then Step 6.1 (answer-merge).

6. Merge proposals into concept/entity pages (locked, claim-dedup)

concept-store.py merge parses the records, derives each slug via slugify(title), and under cogni-wiki's _wiki_lock creates-or-merges each page with claim-level dedup. It owns all serialization + the created-vs-updated decision + a pre-write round-trip self-check:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/concept-store.py merge \
    --records <project_path>/.metadata/distill-records.txt \
    --wiki-root <WIKI_ROOT> \
    --project-path <project_path> \
    --project-slug <project-slug> \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS" \
    --bundle-hash "$SHA"

--bundle-hash "$SHA" (the Step-1 content hash) is written into the manifest by concept-store.py itself — it is the single writer, so Step 3's resume check reads it back with no fragile second-process patch. Parse data: created_slugs[], updated_slugs[] (disjoint), concepts[] (each {slug, type, action, summary, claims_new, claims_deduped, claims_rejected, near_existing_slug, ...}), claims_attached_total, claims_deduped_total, claims_rejected_total, near_existing_total, near_existing_slugs[] ({slug, near_slug, near_title, near_type, score}). On success: false → warn + exit cleanly. If claims_rejected_total > 0, surface it loudly — it means the distiller emitted malformed claim lines (e.g. dropped the <slug> | <id> | provenance), which would otherwise silently shrink the concept web; check the records file format. near_existing_total / near_existing_slugs[] are the title→slug observable tripwire — see Step 9.

6.1. Merge answer claims into each question node (locked, claim-dedup, serialized after Step 6)

The two distiller merges share cogni-wiki's _wiki_lock, so they run serialized — concept-store.py merge (Step 6) first, then this question-store.py answer-merge immediately after, before the cross-lingual (6.6) and re-narrate (6.7) passes. The two agents already ran concurrently in the Step 5 wave; only their merges serialize here. Skip cleanly when ANSWER_Q_COUNT == 0 from Step 4.5, the answer-distiller was not dispatched, or its Step-5 envelope was questions_proposed: 0 / ok: false — report n/a (no answerable question nodes) and jump to Step 6.6.

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/question-store.py answer-merge \
    --records <project_path>/.metadata/answer-records.txt \
    --wiki-root <WIKI_ROOT> \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS"

answer-merge parses the records and, under cogni-wiki's _wiki_lock (a question node is a shared-state read-modify-write of an existing page), splices a deduped answer_claims: block into each node's frontmatter — claim-level dedup keyed on <source_slug>#<claim_id> (exact norm_key → claim_similarity ≥ 0.85, fail-safe keep-both; union backlinks[] + source_claim_refs[], never drop a provenance ref), minting acl-NNN ids, with a pre-write round-trip self-check. It preserves every other frontmatter key, the ## Findings block, and the human ## Notes tail byte-for-byte, and operates only on type: question pages (refusing foundation: true). Parse data: questions[] (each {slug, action ∈ created_block|updated|unchanged|skipped|write_failed, reason, claims_new, claims_deduped, claims_rejected}), claims_attached_total, claims_deduped_total, claims_rejected_total. On success: false → warn + continue to Step 6.6. If claims_rejected_total > 0, surface it loudly (the distiller emitted malformed claim lines — check the records format). Capture the counts for Step 9.

Note on the emit interaction (no code change to emit). A later run's ingest Step 4.5 (question-store.py emit) re-renders a question node's frontmatter from its fixed template, which does not carry answer_claims: — so it drops the block until that run's distill re-adds it. Within a run the order is always ingest → distill, so the surface is restored each run; if distill is skipped (it is optional), the node reverts to framing-only — exactly this phase's fail-soft contract, identical downstream to today.

Fail-soft at every hop. A distiller ok:false, a missing/empty records file, or a non-zero answer-merge exit must each warn and continue to Step 6.6 with the question nodes intact (framing-only). This pass is enrichment, never a gate.

6.6. Cross-lingual claim merge (DE↔EN, default-on, fail-soft, auto-skip)

Phase-1 dedup (Step 6) deliberately under-merges across languages: the only deterministic DE↔EN bridge is the article-number digit anchor (×3.0), so a German claim and its English twin survive as two distilled_claims[] entries. That is the safe direction (a wrong cross-language merge silently destroys a distinct fact and is unrecoverable) but on a mixed-language base (EN+DE sources) it is lossy — a concept page lists each fact twice and the dedup ratio under-reports the real overlap. This step has an LLM confirm which script-flagged candidate pairs are the same fact stated in two languages, then unions them. It NEVER fires on a single-language base — the candidate generator emits nothing there (same-language twins already collapsed in Step 6), so this step self-skips with zero LLM cost (the auto-skip).

Skip cleanly when: --no-crosslingual was passed, OR created_slugs[] + updated_slugs[] (from Step 6) is empty. Jump to Step 6.7.

a. Generate candidates (deterministic, read-only). xlingual-candidates pairs each touched page's distilled_claims[] and flags those that share an article-number digit anchor but did NOT auto-merge (low overlap) — the signature of a DE↔EN twin. Save the envelope, then transform it into the agent's bundle:

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/concept-store.py xlingual-candidates \
    --wiki-root <WIKI_ROOT> \
    --slugs "<comma-separated created_slugs + updated_slugs>" \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS" \
    > <project_path>/.metadata/xlingual-candidates.json

On success: false → warn + continue to Step 6.7 (this step never blocks). Otherwise build the per-pair bundle (env-var paths so quotes can't break the literal):

Run the cross-lingual candidate bundle subprocess in references/distill-bundle-builders.md §3 — env inputs and the verbatim code are there; it prints the count the next paragraph captures.

If the printed candidate count is 0 → skip cleanly to Step 6.7 (the auto-skip; report n/a (no cross-lingual candidates)).

b. Dispatch the merger (single Task call).

Task(cross-lingual-claim-merger,
     CANDIDATES_PATH=<project_path>/.metadata/xlingual-candidates.txt,
     RECORDS_OUTPUT_PATH=<project_path>/.metadata/xlingual-records.txt,
     OUTPUT_LANGUAGE=<plan.json::output_language, default en>)

cross-lingual-claim-merger lives at ${CLAUDE_PLUGIN_ROOT}/agents/cross-lingual-claim-merger.md — dispatched via Task, not Skill. It judges each candidate "same fact, two languages?" and writes a raw-text merge: <slug> | <survivor> | <absorbed> line per confirmed twin. Parse the return: ok: true → continue to 6.6c; ok: false or pairs_confirmed: 0 → warn/continue to Step 6.7 (no unions to apply).

c. Apply the unions (locked, fail-safe).

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/concept-store.py crossmerge \
    --records <project_path>/.metadata/xlingual-records.txt \
    --wiki-root <WIKI_ROOT> \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS"

crossmerge parses the records and, under the SAME _wiki_lock as merge, re-validates the candidate gate server-side for every record (both ids must still be on the page AND the pair must still satisfy the deterministic candidate predicate — so the LLM can never widen scope) and UNIONs the absorbed claim's backlinks + source_claim_refs onto its survivor — never dropping a provenance ref, only removing the duplicate dcl-id. Parse data: merged[] ({slug, survivor_id, absorbed_id}), merged_slugs[], skipped[] ({slug, reason}; reasons page_not_found/no_sentinels_human_page/claim_not_found/not_a_candidate/claims_round_trip_mismatch), n_merged, n_skipped, claims_crossmerged_total. On success: false → warn + continue to Step 6.7.

Fold merged_slugs[] into updated_slugs[] so Step 6.7 re-narrates the now-shorter claim list and Step 7's wiki integration covers the page. Capture n_merged / n_skipped for Step 9.

Fail-soft at every hop. A failed candidate scan, a merger ok:false, a missing/empty records file, or a non-zero crossmerge exit must each warn and continue to Step 6.7 with claims intact. This pass is enrichment, never a gate. (Re-running on the same claim set is byte-stable: an already-absorbed dcl id is gone, so the record re-validates to claim_not_found and the page is untouched.)

6.7. Re-narrate updated summaries (default-on, fail-soft)

Phase 4.5 compounds the ## Claims / ## Related / ## Sources blocks across runs, but concept-store.py merge keeps the ## Summary block first-writer-wins on update — so an updated page can list 20 distilled claims under prose that still reflects only run 1's framing. This step re-narrates the summary of each updated page from its merged claims so the wiki compounds narratively, not just structurally. Summary-only: every other machine block + the human ## Notes tail stay byte-identical.

Skip cleanly when: --no-renarrate was passed, OR updated_slugs[] (from Step 6, possibly grown by Step 6.6's merged_slugs[] fold so crossmerged pages get re-narrated too) is empty (created pages already carry a fresh distiller summary; pure re-runs have no updated pages). In either case jump straight to Step 7 — re-narration is purely additive enrichment.

Otherwise:

a. Build the per-slug bundle. For each slug in updated_slugs[], read its on-disk page, pull the existing ## Summary inner (stripping the ## Summary heading line) + the merged distilled_claims[].text, and append a block to renarrate-bundle.txt. Reuse the shared parsers — no new block parsing:

Run the re-narrate bundle subprocess in references/distill-bundle-builders.md §4 — env inputs and the verbatim code are there; it prints the count the next paragraph captures.

If the printed slug count is 0 → skip to Step 7 (nothing to re-narrate).

b. Dispatch the narrator (single Task call).

Task(concept-summary-narrator,
     RENARRATE_BUNDLE_PATH=<project_path>/.metadata/renarrate-bundle.txt,
     RECORDS_OUTPUT_PATH=<project_path>/.metadata/renarrate-records.txt,
     OUTPUT_LANGUAGE=<plan.json::output_language, default en>)

concept-summary-narrator lives at ${CLAUDE_PLUGIN_ROOT}/agents/concept-summary-narrator.md — dispatched via Task, not Skill. Parse the return envelope: ok: true → continue to Step 6.7c; ok: false → warn (surface the error) and jump to Step 7 with summaries unchanged (re-narration never blocks the pipeline).

c. Apply the re-narration (locked, summary-only).

python3 ${CLAUDE_PLUGIN_ROOT}/scripts/concept-store.py renarrate \
    --records <project_path>/.metadata/renarrate-records.txt \
    --wiki-root <WIKI_ROOT> \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS"

renarrate parses the records, and under the SAME _wiki_lock as merge replaces only the SUMMARY machine block of each named page (every other block + ## Notes byte-identical), bumping updated: only when the prose actually changed. Parse data: renarrated[], unchanged[], skipped[] (each {slug, reason}; reasons page_not_found / no_summary_sentinel). On success: false (e.g. records_not_found, or a --wiki-scripts-dir import failure) → warn + continue to Step 7. Capture n_renarrated / n_unchanged / n_skipped for the Step 9 summary. (Unlike Step 6's merge, renarrate needs neither --project-path nor --project-slug — it accepts them for call-site symmetry but ignores them, so the omission here is deliberate.)

Fail-soft at every hop. A narrator ok:false, a missing/empty records file, or a non-zero renarrate exit must each warn and continue to Step 7 with the existing summaries intact. Re-narration is enrichment, never a gate. (Note: because this runs only for updated_slugs[] and merge already bumped those pages' updated: to today in Step 6, a re-narration date bump lands on the same date — no extra churn.) The two parsed envelopes use different success keys by design: the agent returns ok (agent convention, like concept-distiller), while the concept-store.py renarrate script returns success (the insight-wave script-output convention) — so the dual-key handling above is intentional, not a bug.

7. Per-new/updated-slug cogni-wiki integration (sequential, after merge)

For each slug in created_slugs[] + updated_slugs[], in deterministic order (skip skipped/write_failed/unchanged slugs — they need no index/backlink work):

1. Backlink audit + apply (forms concept↔source / concept↔concept edges). Identical mechanism to knowledge-ingest Step 4.1 — audit, curate a targets[] plan from genuine relations only, apply:

   python3 "$WIKI_INGEST_SCRIPTS/backlink_audit.py" --wiki-root <WIKI_ROOT> --new-page <slug> --top 8 --min-confidence medium
   

   Curate {"targets": [{"slug": "<target>", "sentence": "... [[<slug>]] ..."}]} from genuine relations (skip weak matches; each sentence MUST contain [[<slug>]]), then:

   printf '%s' "$PLAN_JSON" | python3 "$WIKI_INGEST_SCRIPTS/backlink_audit.py" --wiki-root <WIKI_ROOT> --new-page <slug> --apply-plan -
   

   apply_plan is idempotent + fail-soft per target. If no genuine relation, skip apply for that slug (never invent a backlink). The concept page already carries bare [[<source-slug>]] links in its ## Sources block (written by concept-store.py), so its concept→source edges exist; this step adds the inbound source→concept / concept→concept edges.

2. Index update (thematic category). File the page under the category matching its type (from the merge result): --category "Concepts" for type: concept, --category "Entities" for type: entity, --category "People" for type: person. Use the merge result's summary (always non-empty — concept-store.py falls back to the title), but first sanitize it so a stray typographic substitute (U+2020 DAGGER, U+2021, or an exotic space U+00A0/U+202F/U+2009) never reaches the reader-facing wiki/index.md one-liner — same guard knowledge-ingest Step 4.2 applies, pass the raw value via an env var:

   CLEAN_SUMMARY=$(KNOWLEDGE_SCRIPTS="${CLAUDE_PLUGIN_ROOT}/scripts" \
   RAW_SUMMARY="<the page's summary from the merge result>" \
   python3 -c '
   import os, sys
   sys.stdout.reconfigure(encoding="utf-8")
   sys.path.insert(0, os.environ["KNOWLEDGE_SCRIPTS"])
   from _knowledge_lib import sanitize_summary
   print(sanitize_summary(os.environ["RAW_SUMMARY"]))
   ')
   

   Then pass $CLEAN_SUMMARY to --summary. Every flag is on a continued line (trailing \); do not put a shell comment on an argument line, or --max-summary is dropped and the mid-word clamp is lost:

   python3 "$WIKI_INGEST_SCRIPTS/wiki_index_update.py" --wiki-root <WIKI_ROOT> --slug <slug> \
       --summary "$CLEAN_SUMMARY" \
       --category "<Concepts|Entities|People per the merge result's type>" \
       --max-summary 240
   

   Capture the envelope. When success == true and data.action == "inserted", increment n_new (init 0 before the loop). data.action == "updated" (a row already existed) does NOT count — same lockstep as knowledge-ingest Step 4. In the same inserted branch, also increment the per-type insert counter for this page's type — n_new_entities / n_new_people (each init 0 before the loop), keyed off the merge result's type (entity / person). Step 7.1 below gates each sub-index render on its own per-type counter. (There is no n_new_concepts render counter here: wiki/concepts/index.md is rendered by its dedicated concepts_index.py at knowledge-finalize Step 10.5 sub-step 3.6, not by sub_index.py. The total n_new still counts every type — concepts included — so the entries_count bump below is unchanged.)

3. On any helper failure, record in failed_index_updates[] and continue — the page is on disk; only discoverability is incomplete.

After the loop, bump entries_count once by the count of newly-indexed pages (only when n_new > 0), exactly as knowledge-ingest Step 4 / knowledge-finalize Step 8:

python3 "$WIKI_INGEST_SCRIPTS/config_bump.py" --wiki-root "$WIKI_ROOT" --key entries_count --delta <n_new>

Non-fatal on failure (reconcile via wiki-lint --fix=entries_count_drift). Do NOT run the full lint_wiki.py --fix=all / health.py conformance gate here — that stays in knowledge-finalize Step 10.5, which runs it once, at the end, over the page set that now includes these distilled pages. The bounded --fix=reverse_link_missing de-orphan gate in Step 7.2 below is the deliberate exception.

7.1. Render the distilled sub-indexes (wiki/{entities,people}/index.md)

After the per-slug index/backlink loop and the entries_count bump, re-render the machine-owned per-type sub-indexes for the two sub_index.py-owned distilled types (entities, people) so each reflects the pages filed this run. This is the distill-phase sibling of the knowledge-ingest Step 4 sources render and Step 4.5.6 questions render — the same generic deterministic renderer knowledge-setup seeds at bootstrap. Concepts are deliberately excluded: wiki/concepts/index.md is owned by the dedicated concepts_index.py renderer (with CONCEPTS-LEADIN narration) at knowledge-finalize Step 10.5 sub-step 3.6, not by sub_index.py.

Run one render per type, each gated on its own per-type counter (n_new_entities / n_new_people from Step 7 sub-step 2) — a clean re-run that merged every page in place added no new index row for that type, so its sub-index is already current and the render is skipped (idempotent: an unchanged wiki is a no-op). Each render must run after the per-slug wiki_index_update.py --category calls above (which file every new distilled page under its ## <theme> heading in wiki/index.md) — otherwise a just-written page lands in the renderer's trailing ## Uncategorized group. The generic renderer groups each page under its own portal theme (theme_via_own_slug, same as the sources render) and carries any narrator-authored MACHINE-OWNED:ENTITIES-LEADIN:<theme> / PEOPLE-LEADIN:<theme> lead-in forward verbatim (no clobber), so rendering here never overwrites a lead-in a later run narrates:

# Derive the base's output language so the engine-owned per-theme lead-in
# fallback reads in-language on a non-English base (default en; the narrator
# overwrites it on a later run regardless):
OUTPUT_LANGUAGE=$(python3 -c '
import json, sys
try:
    print((json.load(open(sys.argv[1])).get("output_language")) or "en")
except Exception:
    print("en")
' "<project_path>/.metadata/plan.json")
# One block per type; run only when that type's per-type counter > 0.
# entities:
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/sub_index.py" render \
    --type entities \
    --wiki-root "$WIKI_ROOT" \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS" \
    --lang "$OUTPUT_LANGUAGE"
# people:
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/sub_index.py" render \
    --type people \
    --wiki-root "$WIKI_ROOT" \
    --wiki-scripts-dir "$WIKI_INGEST_SCRIPTS" \
    --lang "$OUTPUT_LANGUAGE"

Fail-soft — a renderer failure never rolls back distill: every distilled page, backlink, index row, and entries_count bump is already on disk. The render is lock-wrapped (_wiki_lock) + atomic (atomic_write_text) at its own write site and writes only when the proposed text differs byte-for-byte, so a forced failure leaves no partial page. sub_index.py is itself fail-soft (a missing wiki-scripts dir, a _wikilib import failure, or a non-wiki --wiki-root returns an error envelope rather than raising), so the orchestrator treats a non-zero result as a surfaced warning, never an abort. Surface each per-type outcome in the Step 9 summary and continue.

7.2. Bounded de-orphan gate (reverse_link_missing)

The distilled pages just written carry forward [[<source-slug>]] edges (concept→source) in their ## Sources block, but the source pages do not yet hold the reverse [[<concept-slug>]] link — so a standalone distill (run later on an already-finalized base, with no knowledge-compose → knowledge-finalize to follow) would leave every page this phase wrote as an orphan_page with a reverse_link_missing gap until some future finalize that may never come. knowledge-ingest does not have this problem because it de-orphans its own pages inline (Steps 4.1 / 4.5.2); distill mirrors that posture here with a bounded, idempotent gate — not the whole-run conformance gate, only the one load-bearing reverse-link backfill.

Resolve the cogni-wiki wiki-lint scripts dir with the SAME resolve_wiki_scripts helper used at the top of this skill — fail-soft: on a miss, warn and continue (the base self-heals on the next knowledge-finalize / knowledge-lint), exactly as the wiki-ingest resolution does:

. "${CLAUDE_PLUGIN_ROOT}/scripts/resolve-wiki-scripts.sh"
WIKI_LINT_SCRIPTS=$(resolve_wiki_scripts wiki-lint lint_wiki.py) || { echo "⚠ cogni-wiki wiki-lint scripts not found — skipping the bounded de-orphan gate (run knowledge-finalize or knowledge-lint to reconcile)"; WIKI_LINT_SCRIPTS=""; }

When resolved, run the single bounded fix class. It mirrors every distilled page's ## Sources [[<source-slug>]] edge back onto the source (and onto any cited wiki/questions/<slug>.md node), giving each page this phase wrote an inbound link. It is idempotent — a no-op on an already-clean base — and writes nothing else (--fix=all's other four classes stay in finalize):

[ -n "$WIKI_LINT_SCRIPTS" ] && python3 "$WIKI_LINT_SCRIPTS/lint_wiki.py" \
    --wiki-root "$WIKI_ROOT" \
    --fix=reverse_link_missing

Capture the envelope; surface data.fixed[] / data.failed[] counts in the Step 9 summary. Non-fatal per item — a per-page fix failure lands in data.failed[] and never blocks the phase. (orphan_page is a lint warning, not a --fix class; 0 orphans comes from the inbound links this reverse-link backfill writes, never from --fix directly.)

8. Append wiki/log.md

DATE_STAMP=$(date -u +%F)
TOPIC=<topic from plan.json>
N_CONCEPTS=<len(created_slugs) + len(updated_slugs)>
N_ATTACHED=<claims_attached_total>
N_DEDUPED=<claims_deduped_total>
LOG_PATH=$(python3 "${CLAUDE_PLUGIN_ROOT}/scripts/control-path.py" log --wiki-root "${WIKI_ROOT}")
echo "## [${DATE_STAMP}] distill | project=${TOPIC} concepts=${N_CONCEPTS} claims=${N_ATTACHED} deduped=${N_DEDUPED}" >> "${LOG_PATH}"

The distill prefix is additive-safe — cogni-wiki readers bucket an unknown prefix in their catch-all without crashing (same posture compose/verify/finalize had before cogni-wiki formalized them).

9. Final summary

Print ≤ 12 lines:

• Project: <topic> at <project_path>
• Wiki: <WIKI_ROOT>
• Distilled pages created: <n> / updated: <n> / unchanged: <n> / skipped: <n> (reasons: foundation_collision/no_sentinels_human_page/slug_type_collision/empty_slug)
  • By type (created): concepts <c> / entities <e> / people <p> — count each from the merge result's per-slug type (omit a type's tally when it is 0)
• Claims attached: <claims_attached_total> (deduped: <claims_deduped_total> → dedup ratio <deduped/attached>); if claims_rejected_total > 0, add ⚠ <claims_rejected_total> claim lines rejected as malformed — check the distiller's records format
• Cross-lingual merges: <n_merged> (<n_skipped> skipped) — or skipped (--no-crosslingual) / n/a (no cross-lingual candidates) when Step 6.6 did not fire (the single-language norm — no DE↔EN twins to merge)
• Summaries re-narrated: <n_renarrated> (<n_unchanged> unchanged, <n_skipped> skipped) — or skipped (--no-renarrate) / n/a (no updated pages) when Step 6.7 did not run
• Question nodes answered: <n with answer_claims created/updated> (claims attached <answer_claims_attached_total>, deduped <answer_claims_deduped_total> — these are the answer-merge envelope's claims_attached_total/claims_deduped_total/claims_rejected_total, distinct from the Step-6 concept-store counts above) — or n/a (no answerable question nodes) when Step 6.1 did not fire; if the answer-merge claims_rejected_total > 0, add ⚠ <n> answer-claim lines rejected as malformed — check the answer-distiller's records format
• title→slug tripwire — if near_existing_total > 0, surface a warning block:
  • Header: ⚠ <near_existing_total> concepts created near an existing slug — check title stability
  • One line per entry from near_existing_slugs[] (deterministic order, score-sorted desc): <slug> ~ <near_slug> (<near_type>, score=<score>)
  • Subline: If these are the same concept, the run forked a near-duplicate page; rename the proposal in the next run, or merge manually via the wiki.
  • When near_existing_total == 0 print nothing (no false-alarm noise on clean runs).
• Wiki entries_count: +<n_new> (or ⚠ bump failed — run wiki-lint --fix=entries_count_drift; or unchanged when n_new == 0)
• Sub-indexes rendered (Step 7.1): entities / people — per type, re-rendered (counter > 0), skipped (no new rows) (counter == 0), or ⚠ render failed — <reason> (fail-soft); concepts are rendered separately by knowledge-finalize sub-step 3.6
• Reverse-link backfill (Step 7.2): <n_fixed> link(s) added (<n_failed> failed) — or 0 (already clean) / skipped (wiki-lint scripts not found) when the bounded de-orphan gate found nothing to do or could not resolve its scripts dir
• Cost: $X.XXX (from the distiller return)
• Next: knowledge-compose reads the distilled pages (concept/entity today; person joins the compose read surface when its compose/verify wiring lands) as framing context (not citable evidence).

The dedup ratio is the compounding success metric (differentiation-thesis.md): of the new facts proposed this run, the fraction that merged into an existing claim instead of adding a duplicate line.

The title→slug tripwire is pure observability — it never blocks the pipeline, never auto-merges, never skips a write. A near_existing_slug warning means claim_similarity(new_title, existing_title) >= 0.65 on the symmetric weighted-Jaccard primitive; titles in that band MAY be a silent slug-fork (e.g. "Hochrisiko-Klassifizierung" vs "Einstufung als hochriskant" — different slugs, same concept) but may also be genuinely-distinct neighbours. Human judgment owns the disposition.

Edge cases

• Fresh base (run 1). Every concept/entity is created; claims_deduped_total may be 0 (nothing prior to dedup against — same-run near-dupes still merge). Compounding shows from run 2 onward.
• Re-run on the same claim set. Step 3 skips re-dispatch when bundle_hash is unchanged; even without the skip, concept-store.py merge is byte-stable (re-merge → all unchanged, n_new == 0, no bump); updated_slugs[] is then empty, so Step 6.7 re-narration also no-ops.
• A concept slug collides with a foundation: true page. concept-store.py refuses to merge (skipped, reason: foundation_collision) — never overwrites a curated foundation.
• A concept slug collides with a hand-authored page (no MACHINE-OWNED sentinels). concept-store.py skips it (reason: no_sentinels_human_page) and leaves the page untouched — we never clobber a page we did not author.
• Distiller proposed a concept but every claim was a re-run duplicate. The page is unchanged; no index churn, no entries_count bump.
• Empty / claim-less sources. Sources with no pre_extracted_claims: are omitted from the bundle; if the whole bundle is empty, the phase no-ops cleanly.
• No question nodes / base predates the question-node feature. Step 4.5 finds no wiki/questions/ dir (or no node with answerable claims), so ANSWER_Q_COUNT=0 — the Step 5 wave drops the answer-distiller leg and Step 6.1 self-skips cleanly (n/a (no answerable question nodes)) — no answer_claims: work, no error.
• Distill skipped entirely. Question nodes keep only their ## Findings + ## Notes (framing-only) — byte-identical to behavior before the answer surface existed; the composer reads them framing-only either way (the citable path is a later activation step).
• Re-ingest before re-distill. A later run's ingest emit re-renders the question frontmatter without answer_claims: (it has no such template field); the same run's distill (Step 6.1) re-adds it. Transiently framing-only between ingest and distill, restored by distill — never a lost-data state for a completed run.
• Single-language base (the norm). Step 6.6's xlingual-candidates finds no pairs (same-language twins already collapsed in Step 6), so the cross-lingual pass self-skips with zero LLM cost — n/a (no cross-lingual candidates). The feature only does work on a mixed DE↔EN base.

Out of scope

• Does NOT compose the draft — that is Phase 5 (knowledge-compose).
• Re-narrates the ## Summary body of updated distilled pages (any of the three types — concept / entity / person) from the merged claims (Step 6.7, default-on, fail-soft; --no-renarrate opts out). created pages keep the distiller's fresh summary; pure re-runs touch nothing. It does NOT re-synthesize any other block, and it does NOT add a contradiction pass.
• Merges cross-lingual (DE↔EN) twin claims on a mixed-language base (Step 6.6, default-on, fail-soft, auto-skip; --no-crosslingual opts out). An LLM only confirms pairs the script flagged (shared article-number anchor + low overlap); concept-store.py crossmerge re-validates the gate and UNIONs provenance onto the survivor — never dropping a fact. It does NOT touch single-language dedup (Step 6's job), and it explicitly does NOT use embedding/vector similarity (approach (c), rejected by the differentiation thesis).
• Emits three page types — concept / entity / person (named humans, split out of entity). It does NOT emit any other cogni-wiki page type (sources are Phase 4, syntheses are Phase 7).
• Does NOT run the full lint_wiki.py --fix=all / health.py whole-run conformance gate — knowledge-finalize Step 10.5 covers the whole run once. It DOES run a bounded --fix=reverse_link_missing de-orphan gate inline (Step 7.2) so a standalone distill leaves the base structurally clean (0 orphans, 0 reverse-link gaps), mirroring knowledge-ingest's inline posture.
• Does NOT modify binding.json — Phase 7 (knowledge-finalize) appends the project entry.
• Does NOT block the pipeline — every failure path warns and exits cleanly.

Output

• <WIKI_ROOT>/wiki/{concepts,entities,people}/<slug>.md — created or enriched per the proposal's type:, with distilled_claims: frontmatter, MACHINE-OWNED body sentinels, and bare [[<source-slug>]] backlinks. A human ## Notes region is preserved byte-for-byte across runs.
• <WIKI_ROOT>/wiki/index.md — each page filed under ## Concepts / ## Entities / ## People.
• <WIKI_ROOT>/wiki/entities/index.md — re-rendered (Step 7.1, when n_new_entities > 0) — the machine-owned entities sub-index, grouped by portal theme via sub_index.py render --type entities; narrator-authored ENTITIES-LEADIN spans are carried forward verbatim.
• <WIKI_ROOT>/wiki/people/index.md — re-rendered (Step 7.1, when n_new_people > 0) — the machine-owned people sub-index, grouped by portal theme via sub_index.py render --type people; narrator-authored PEOPLE-LEADIN spans are carried forward verbatim.
• Existing pages gain curated [[<slug>]] inbound backlinks (via backlink_audit.py --apply-plan).
• <WIKI_ROOT>/.cogni-wiki/config.json — entries_count bumped by <n_new>.
• <WIKI_ROOT>/wiki/log.md — one new ## [YYYY-MM-DD] distill | … line.
• <project_path>/.metadata/distill-manifest.json (schema 0.1.1) + intermediate distill-bundle.txt / distill-slug-index.txt / distill-records.txt; plus (when Step 6.6 fires) xlingual-candidates.json / xlingual-candidates.txt / xlingual-records.txt; plus (when Step 6.7 runs) renarrate-bundle.txt / renarrate-records.txt; plus (when the answer path runs — Step 4.5 build + Step 6.1 merge) answer-bundle.txt / answer-records.txt.
• Updated distilled pages (any of the three types — concept / entity / person) get their ## Summary body re-narrated from the merged claims (Step 6.7); all other machine blocks + the ## Notes tail stay byte-identical.
• <WIKI_ROOT>/wiki/questions/<slug>.md — each type: question node gains/enriches an answer_claims: frontmatter block (Step 6.1, acl-NNN ids, claim-deduped, with backlinks[]/source_claim_refs[] provenance); the ## Findings block and the human ## Notes tail stay byte-identical.

References

• ${CLAUDE_PLUGIN_ROOT}/references/inverted-pipeline.md — Phase 4.5 contract
• ${CLAUDE_PLUGIN_ROOT}/references/differentiation-thesis.md — the compounding loop + claims-dedup metric this phase realizes
• ${CLAUDE_PLUGIN_ROOT}/agents/concept-distiller.md — dispatched agent (Phase 1 proposals)
• ${CLAUDE_PLUGIN_ROOT}/agents/concept-summary-narrator.md — dispatched agent (Step 6.7 summary re-narration)
• ${CLAUDE_PLUGIN_ROOT}/agents/cross-lingual-claim-merger.md — dispatched agent (Step 6.6 cross-lingual DE↔EN claim merge)
• ${CLAUDE_PLUGIN_ROOT}/agents/answer-distiller.md — dispatched agent (Step 5 fan-out wave answer-claim synthesis for question nodes)
• ${CLAUDE_PLUGIN_ROOT}/scripts/concept-store.py --help — locked create-or-merge + claim-dedup engine (incl. xlingual-candidates / crossmerge)
• ${CLAUDE_PLUGIN_ROOT}/scripts/question-store.py --help — answer-merge (Step 6.1 locked answer_claims: splice) + emit (Phase 4 question nodes)
• ${CLAUDE_PLUGIN_ROOT}/scripts/knowledge-binding.py --help