distill-flow

name

distill-flow

description

Gradient-descent-style distillation for any content. Runs a distill→score→refine loop that compresses a source — documents, transcripts, code, logs, prompts, other skills, or a PERSON/AGENT's knowledge + personality + "soul" — to a target level while holding fidelity high, then reports the quality-vs-compression trade-off and stops at the knee of the Pareto frontier. Use whenever the user wants to distill, condense, compress, summarize-without-losing-fidelity, shrink, or tighten something, and ESPECIALLY when they care about the cut-vs-keep trade-off, mention a length/token budget, ask for a high-fidelity summary, or want the sweet spot between brevity and completeness. ALSO trigger for distilling a person, colleague, character, or agent into a persona that behaves and sounds like them (voice, decisions, values/soul graded via behavioral probes), for distilling prompts/context-packs/SKILL.md files, and for adding a new distillation type when no built-in one fits. Pairs with skills-flow.

DistillFlow

Gradient descent for distillation.

Most "summarize this" attempts pick a length out of the air and hope the result kept the important parts. DistillFlow refuses to guess. It treats distillation as a two-objective optimization — maximize compression, maximize fidelity — runs an execute→evaluate→refine loop against a probe set that defines what "fidelity" concretely means for this source, and reports the Pareto frontier so the human (or the skill itself) can stop at the knee: the point where squeezing out more length starts costing real meaning.

The whole pitch: a candidate distillation is the weights, the source plus a must-preserve probe set is the training data, a probe-recall score is the loss, and a Refiner that recovers dropped facts without inflating length is the optimizer. The loop traces the trade-off curve instead of climbing one number blindly.

---

The analogy (ported from skills-flow)

ML / skills-flow	DistillFlow equivalent
Model weights / the skill	The candidate distillation (the condensed text)
Training dataset	Source material + a probe set (the must-preserve rubric)
Loss function	Two objectives: probe-recall fidelity and compression ratio
Optimizer / Refiner	Re-distiller that recovers failed probes within the token budget
Forward pass / execution	Produce a candidate at a target level
Backprop / grad step	Refiner reads which probes failed and patches the distillation
Validation / held-out split	Held-out probes the distiller never sees — guards against a summary that reads well but quietly drops facts
Checkpoints v0,v1…	Distillation levels L0→Ln along the frontier
Target pass rate	Target compression at a fidelity floor (the knee)

The discipline carries over: pick the winning level by its held-out fidelity, never its train fidelity. A distillation that scores high only on the probes the distiller could see is "overfit" — it kept the facts it was told to keep and silently dropped the rest. DistillFlow will not select it.

---

The two objectives, and why the knee matters

Compression and fidelity trade off. Plot every candidate as a point (compression, fidelity) and the good ones form a frontier:

fidelity
  1.0 |x x x x x.
      |          'x  <- knee: last point before fidelity falls off
      |             \
      |              \x
      |                \
      |                 'x x   <- over-compressed: cheap length, expensive meaning
  0.0 +-------------------------- compression
      0%                      99%

• Below the knee (less compression): you're leaving easy wins on the table — more can be cut with almost no fidelity loss.
• At the knee: the happy balance. Maximum compression for negligible fidelity loss. This is the default selection.
• Past the knee: each extra bit of compression now costs disproportionate fidelity. Only go here if the user explicitly accepts the loss for a hard budget.

Trade-off awareness is the headline feature. DistillFlow never just returns "here's a shorter version." It returns the chosen level and the frontier behind it, e.g. "At 85% reduction you keep 0.97 fidelity; pushing to 92% drops you to 0.81. The knee is 85% — that's what I selected. Want the tighter one anyway?"

---

The loop

  source + probe set (held-out split reserved)
            │
   ┌────────▼──────────────────────────────────────────┐
   │ For each target level L (e.g. 50/70/85/92% cut):    │
   │                                                     │
   │  1. Distill   produce a candidate within L's token  │
   │     (execute) budget                                │
   │                                                     │
   │  2. Score     answer probes using ONLY the candidate│
   │     (loss)    → fidelity = weighted probe recall    │
   │               (train probes + held-out probes,      │
   │                scored separately)                   │
   │                                                     │
   │  3. Refine    Refiner reads the FAILED probes →     │
   │     (grad)    rewrites to recover them WITHOUT      │
   │               exceeding the budget (swap, don't add)│
   │                                                     │
   │  4. Repeat 2–3 until fidelity stops improving       │
   │     (patience) or the budget can't hold the floor   │
   └────────┬────────────────────────────────────────────┘
            │  collect one (compression, fidelity) point per level
   ┌────────▼────────────────────────────────────────────┐
   │  Frontier   compute Pareto frontier + detect knee     │
   │  Select     knee, subject to the fidelity floor       │
   │  Report     chosen level + the full trade-off curve   │
   └───────────────────────────────────────────────────────┘

---

How to run it

Step 0 — Pick the domain profile

What's being distilled changes what "fidelity" means. Read the matching profile before authoring probes — each one lists the things that must survive and the classic ways distillation silently breaks them:

• Long-form docs, reports, transcripts, threads → references/docs-transcripts.md
• Code, logs, API docs, technical context → references/code-technical.md
• Prompts, context packs, other SKILL.md files → references/prompts-skills.md
• A person or agent — their knowledge, personality, and "soul" → references/persona-soul.md

If the source mixes types (a design doc full of code blocks), read both profiles and merge their probe checklists. If no profile fits — the thing that counts as fidelity isn't fact-recall, voice, or behavior but something else (enforceability, reproducibility, playability…) — do not force-fit the nearest one. Generate a new type: read references/extending.md and scaffold one with python -m scripts.new_profile --slug <type> --what "<what it distills>". The scorer and frontier are type-agnostic, so a new type needs only a profile file and probes built to its criteria.

Step 0.5 — Pick the distillation strategy

The domain decides what counts as a fact; the strategy decides which facts you're trying to keep. They're orthogonal — a code source can be distilled selectively or spectrum-wide. Strategy changes the prompt, the loss, and where the knee sits, so declare it up front. Read references/strategies.md for the full taxonomy; the two main strategies:

Strategy	Goal	Fidelity =	Knee tends to be
selective	Preserve one focus/query/aspect; drop everything off-target	Recall of the targeted probes only (off-target is free to cut)	Deep — most of the source is droppable
spectrum-wide	Preserve breadth; compress every region proportionally	Coverage: per-region recall, penalizing any collapsed region	Shallow — every region must stay present

Selective needs a target (the query/goal/aspect). Spectrum-wide needs probes that span every region of the source, each tagged with its region, so the scorer can catch a region quietly vanishing. Other techniques (hierarchical / multi-resolution, extractive-vs-abstractive, lossless-structural) are knobs that layer on top — see the reference. If unsure, default to spectrum-wide: it's the safe, coverage-preserving choice; switch to selective only when there's a clear single focus.

Step 1 — Build the probe set

The probe set is the fidelity definition. Without it you're back to guessing. Read references/dataset-format.md for the schema. In short, a probe is a fact/claim/question the source answers, plus how to check it survived:

• Deterministic probes — checkable by string/regex/number presence (a named entity, a figure, an API signature, a decision). Scored by scripts/fidelity.py with no model needed.
• Semantic probes — require judgment ("does the distillation still convey that the rollback was caused by the migration, not the deploy?"). Graded by a model.
• Behavioral probes — for persona/agent sources: a situation plus a rubric for the in-character response. Graded by instantiating the distilled persona and judging how it reacts. This is how "soul" — values and decision patterns — is measured, since it only shows under a decision. See references/persona-soul.md.

Auto-extract a first draft of probes from the source (entities, numbers, headings, signatures, decisions), then let the user add any must-keeps. Reserve ~30–40% of probes as held-out — the distiller and Refiner never see these; they only score.

Step 2 — Detect the environment and run the loop

DistillFlow runs the same loop two ways. Auto-detect which you're in:

• Claude Code / agent runtime (a claude CLI is on PATH and the network/runtime allows subprocesses): run it automated.

  python -m scripts.distill_loop \
    --source path/to/source.md \
    --probes probes.json \
    --strategy spectrum-wide \
    --levels 50,70,85,92 \
    --fidelity-floor 0.9 \
    --patience 2 --max-iters 4
  

  The loop shells to claude -p for the distill and refine steps and to grade semantic probes; deterministic probes and all frontier math are pure Python. Strategy comes from --strategy or the probe set's strategy field. For spectrum-wide, score on coverage (--metric coverage on the frontier) so a collapsed region shows as a cliff; for selective, score on targeted recall.

• Claude.ai / no subprocess available: you are the executor and the optimizer. Run scripts/distill_loop.py --plan-only to get the level budgets, the probe split, and an empty scoreboard. Then, by hand: distill to each budget, fill the scoreboard, and call scripts/fidelity.py and scripts/frontier.py (both pure Python, no model calls) to keep score and find the knee. The loop is identical; you play claude -p.

If you can't tell, prefer --plan-only and do it in-session — it always works.

Step 3 — Refine against failures, not vibes

When fidelity misses, do not just rewrite the whole thing. Look at which probes failed (the scorer attributes the miss to specific dropped facts) and recover exactly those — trading out lower-value text so the token budget holds. Adding length to fix a probe is cheating: it moves the point left on the curve instead of up. Recover within budget or report that the budget can't hold the floor.

Step 4 — Report the trade-off, select the knee

End every run with the frontier, not just the winner. Use this shape:

## Distillation report: <source>
Selected level: L=<n> — <compression>% smaller, fidelity <score> (held-out <score>)

Trade-off curve:
  50% cut → fidelity 1.00 (held-out 0.99)
  70% cut → fidelity 0.99 (held-out 0.97)
  85% cut → fidelity 0.97 (held-out 0.95)   ← knee (selected)
  92% cut → fidelity 0.81 (held-out 0.74)   ← past the knee, not selected

Why this level: last point holding above the 0.90 floor with negligible loss.
Dropped past here: <the specific facts that fall out beyond the knee>.

Always name what gets lost past the knee so the human can overrule with eyes open. If they want a tighter level for a hard budget, hand them the past-knee candidate and its dropped-facts list.

---

Guardrails

• Held-out is sacred. Never show held-out probes to the distiller or Refiner, and never select a level on train fidelity. A big train/held-out gap means the distillation memorized the rubric — report it and distrust the level.
• Swap, don't pad. Refinement recovers facts within the budget. If a probe can only pass by growing the output, the honest answer is "this floor isn't reachable at this compression," not a bigger summary.
• Fidelity floor beats compression. When in doubt, select the lower-compression point that clears the floor. Cutting is cheap; lost meaning is expensive.
• Don't fabricate to pass a probe. A probe passes only if the source actually supported it. Distillation never invents.
• Determinism. Pass --seed so a rerun reproduces the same frontier.

---

Files

• references/strategies.md — distillation strategies (selective, spectrum-wide, hierarchical, extractive/abstractive) and how each reshapes probes, loss, and knee
• references/persona-soul.md — profile for distilling a person/agent: knowledge + the 5-layer persona, behavioral/voice/values criteria, ethics
• references/extending.md — how to mint a NEW distillation type when none fits
• references/dataset-format.md — probe-set schema, held-out split, authoring tips
• references/docs-transcripts.md — fidelity profile for long-form content
• references/code-technical.md — fidelity profile for code / logs / API docs
• references/prompts-skills.md — fidelity profile for prompts and other skills
• scripts/fidelity.py — score a candidate against probes (deterministic; emits semantic probes for a judge). Pure stdlib.
• scripts/frontier.py — Pareto frontier + knee detection. Pure stdlib.
• scripts/distill_loop.py — the engine. Auto-detects runtime; --plan-only for in-session use. Pure stdlib (shells to claude -p only when automating).
• scripts/new_profile.py — scaffold a new distillation type's profile. Pure stdlib.
• examples/meeting-notes/ — a tiny runnable example (source + probes).

How it pairs with skills-flow

skills-flow trains a skill to a pass rate; DistillFlow distills content to a trade-off knee. They share the execute→evaluate→refine spine and the held-out-selection discipline. A natural combo: use DistillFlow to compress a bloated SKILL.md or context pack (the prompts-skills profile), then use skills-flow to confirm the slimmed skill still hits its scenario pass rate.