skill-distillation

# Skill Distillation ## Purpose Extract durable procedures, skill updates, or new skills from a completed request loop before delivering the final answer to the user. **Disk writes:** this skill may propose edits to `SKILL.md`, `scripts/`, and related skill-library files. Do **not** write those files until the user confirms the proposed patch (or explicitly asks you to apply it). Default to a dry-run: show the diff/proposal first. ## Prerequisites - Built Isaac Sim (`$ISAAC_SIM_DIR` or `_build/linux-x86_64/release`). - NVIDIA GPU with a current driver (`nvidia-smi`). - Shell env contract from `isaac-sim-orchestrator`: `$ISAAC_SIM_DIR`, `$ISAAC_LAB_DIR`, `$WORKSPACE_DIR`. - User confirmation before any skill-library write (see Write step). ## Limitations - Captures procedures, not automatic commits; human review still applies. - Skip distillation for trivial one-line fixes. - Triggers below mean "propose distillation", not "write silently". ## Troubleshooting | Error / symptom | Cause | Solution | |---|---|---| | Extension or import not found | Wrong `$ISAAC_SIM_DIR` or stale build | Point env vars at `_build/linux-x86_64/release` or rebuild | | Black or empty frames | Missing lights or non-RTX render mode | Add dome/key light; confirm RTX / PathTracing settings | | Hang on stage load or first render | MDL compile or oversized stage | Follow isolation steps in `isaac-sim-troubleshooting` | Step 5 of the universal request loop (see `AGENTS.md` and `meta-skills`). Every user request ends with a distillation pass before delivery. Skip it and the library decays; the next session rediscovers the same lessons. ``` ORIENT -> PLAN -> EXECUTE -> VALIDATE -> DISTILL -> DELIVER ``` ## When to run Propose a distillation pass when any of these happen (still require user confirmation before writing): 1. A task required > 3 iterations **and** the user acknowledged a correction or accepted a working approach. 2. The user corrected you ("that's wrong", "you missed X", "it should be Y"). 3. You discovered a workaround not in any existing skill. 4. A sub-agent hit a failure not covered by an existing skill. 5. You solved something that would break again next session without docs. Also consider during heartbeats (every 2-3 days): review recent `memory/` files for undistilled lessons, then propose updates for confirmation. ## Loop ``` INTERACTION -> TRIGGER CHECK -> EXTRACT -> CLASSIFY -> WRITE -> VERIFY ``` ### 1. Trigger check Ask: - Did I learn something not in any skill file? - Did the user correct my approach? - Did I waste time on something a future agent would also waste time on? - Was there a procedure that worked but isn't documented? Any yes -> extract. ### 2. Extract | What happened | What to extract | |---|---| | Debugging session found root cause | Decision tree (symptom -> diagnosis -> fix) | | User said "do it this way instead" | Procedure update or new rule | | Tried 5 approaches, only 1 worked | Anti-patterns + the working pattern | | Used a tool in a new way | New workflow or technique | | Combined skills in a novel way | Integration pattern | | Task required a specific order | Phase-gated procedure | ### 3. Classify ``` New knowledge ├── Fits existing skill -> UPDATE that skill │ ├── New rule -> Key Rules │ ├── New failure mode -> Failure Modes │ ├── New procedure -> Key Workflows │ └── Correction -> fix in place ├── Cross-cutting pattern -> UPDATE meta-skills ├── Entirely new capability -> CREATE new skill (MSF Phase 1-3) └── One-off fact -> MEMORY.md, not skills ``` Rule: if the knowledge helps solve a different future problem, it belongs in a skill. If it's specific to this one situation, it belongs in memory. ### 4. Write (confirmation required) - Dry-run first: show the proposed skill edit (path + section + patch summary) to the user. - Write only after explicit confirmation (or an explicit user request to apply the update). - Skill updates: edit the specific section; bump the Iteration Log. - New skills: `skills/<name>/SKILL.md` from the meta-skilling template. - Procedure changes: update the orchestrator or relevant workflow skill. - Always: date and context ("Learned from SO-101 session 2026-04-08"). ### 5. Verify - Re-read the file. Does it stand on its own? - Could a fresh agent follow it? - Is the lesson a procedure, not a fact? ## What good distillation looks like Bad: "The table had dual RigidBodyAPI which made it explode." This is a fact about one table; useless next session. Good: "Before using any USD asset in physics, run the Asset Stability Check: load in isolation -> simulate 2 s -> if unstable, scan child meshes for rogue `RigidBodyAPI`." Reusable procedure. Bad: "SO-101 needs hybrid IK + joint-space." Specific to one robot. Good: "Arms with < 6 DOF cannot do lateral transport with pure differential IK. Use hybrid: IK for precision positioning, joint-space for large moves." Generalizes to any small-DOF arm. ## The Generalization Rule Always ask: *"What's the general principle behind this specific fix?"* | Specific fix | Generalized skill | |-------------|-------------------| | Stripped RigidBodyAPI from table legs | Asset stability check procedure | | Used file writes instead of print() | Headless debugging: stdout is unreliable | | Added PhysxArticulationAPI.Apply() | Articulation setup checklist | | Robot moved but didn't grasp | Visual validation must confirm task outcome, not just motion | | User had to give detailed plan | Auto-decomposition: agent breaks down goals into phases | ## Integration with Orchestrator The orchestrator's workflow should include distillation as a final step: ``` Phase 1: Verify Foundations Phase 2: Incremental Integration Phase 3: Polish & Deliver Phase 4: Distill ← NEW (propose after delivery; write only with user confirmation) - What went wrong during this task? - What workarounds were discovered? - What feedback did the user give? - Which skills need updating? - Should a new skill be created? ``` ## Anti-patterns - Waiting to be asked. If you learned it, propose the update now (then confirm before write). - Capturing facts instead of procedures. "X broke" is not a skill; "check X before running" is. - Over-generalizing from a single data point unless the user confirmed the pattern. - Under-generalizing: writing a per-asset fix when the pattern applies to all USD assets. - Skipping verification. An unreadable update is worse than none. - Duplicating across skills. One canonical location per procedure; cross-reference from others. - Embedding code in skills. See below. ## Code in Skills — Reference, Don't Embed Skills should reference canonical sources rather than embed code directly. Embedded code becomes stale when the upstream library or API changes — silently giving future agents wrong patterns to follow. ### Rule For any code pattern in a skill: - *If it comes from Isaac Sim examples or docs:* link to the doc page and/or local example file path. Describe the pattern in prose; let the agent read the actual file. - *If it's a custom utility not in any upstream source:* put it in a versioned script file in the workspace (e.g., `isaac-sim/utils/grasp_utils.py`), reference it by path, and note the version it was written for. - *If it's a tiny one-liner or config snippet* (e.g., a settings dict): embedding is acceptable, but add a comment with the Isaac Sim version it was validated against. ### Why - Isaac Sim APIs change between versions. A skill with embedded code can silently guide agents down wrong paths. - Local example files (`$ISAAC_SIM_DIR/source/standalone_examples/...`) always reflect the installed/built version — they are the most accurate reference available. - Docs at `docs.isaacsim.omniverse.nvidia.com` are version-tagged — link to `latest` unless pinning to a specific release is intentional. ### What to Write Instead of Code - The *pattern name* (e.g., "damped least-squares IK") - *Where to find the canonical implementation* (doc URL + local file path) - *Key parameters* and how to tune them (table form is fine) - *What can go wrong* and how to detect it - *The conceptual flow* in prose: what the code does step by step, without being the code itself ## Library-Health Check (Run During Distillation) Distillation isn't only "add new knowledge." Each pass should also detect when an *existing* skill has grown stale, bloated, or duplicate. Catch these at write time or they accrete until a full consolidation pass is needed. ### Bloat detection Before saving a skill update, eyeball the target file: | Cue | Action | |---|---| | Body > **500 lines** after your edit | Offload to sidecar (see "Sidecar Offload" below). Anthropic spec hard limit. | | Description > **1024 chars** | Compress; move detail to the body. Hard limit. | | ≥ 50% of the file is fenced code blocks | Extract `≥ 20-line` blocks to `scripts/<name>.py`. | | 5+ dated sections (`## Learned 2026-MM-DD`, `## Lessons …`) | Move them to a `lessons.md` sidecar. The main file should describe the *current* procedure, not the journey. | | You're adding a new "Patch 2026-MM-DD" section to fix earlier guidance | **Don't.** Edit the original section in place. Add a 1-line iteration log entry. | ### Sidecar Offload (when SKILL.md is too long) | What | Where | |---|---| | Reusable Python helpers (≥ 20 lines) | `scripts/<name>.py` — import & call from inline 1-liners | | Long worked examples / case studies | `examples.md` | | Lessons / dated discoveries / iteration notes | `lessons.md` | | Deep API reference, parameter tables, config schemas | `reference.md` or `<topic>.md` | | Multi-step workflows that dominate the file | `workflow.md` | **Rule:** one level deep only. `scripts/foo.py` and `examples.md` are valid; `scripts/utils/foo.py` is not (Anthropic spec: one-level refs). ### Duplication check Before adding to a skill, search the library for the same concept: ``` grep -rE "<your-new-concept>" --include=SKILL.md skills/ ``` If another skill already documents it: - **Add to the existing skill**, not your current one. Single source of truth. - **Cross-link** from your current skill: "See `<other-skill>` for X." - If the existing skill's coverage is wrong/outdated → fix it there, don't fork. ### Staleness check Each distillation, ask of the skill you're touching: - Is the framework version reference still current? (Isaac Sim 5.x → 6.0+, PhysX → Newton) - Are the hardcoded paths really paths, or should they be `$ENV_VARS`? - Are there time-sensitive phrases ("before August 2025", "as of 2026-04-04")? Replace with version-anchored or "sample-run" framing. - Does any "lessons learned" section reference a project that no longer exists? ## Cadence | Trigger | Action | |---------|--------| | After every task with user feedback | Immediate distillation pass | | After every task with >3 iterations | Immediate distillation pass | | During heartbeat (every 2-3 days) | Review memory/ for undistilled lessons | | After a PR is merged | Check if merged work revealed patterns worth capturing | | User says "remember this" | Write to memory AND check if it's a skill-level lesson | | New skill created mid-task (no prior skill existed) | Flag as HIGH PRIORITY — see below | | You added content to a skill that now exceeds 500 lines | Offload to sidecar BEFORE delivering response | ## Draft Skills — High Priority Iteration When a skill is created *during* a task (i.e., no existing skill covered the feature), it is by definition unproven. Treat it differently: ### Frontmatter Flag Add to the new skill's YAML frontmatter: ```yaml status: draft priority: high created_from: <task description> ``` ### Behavior While Draft Skill Is Active - *Tell the user immediately:* "I'm working with an untested skill for [feature] — I'll check in after each step rather than at the end." - *Shorten iteration cycles:* share intermediate results after each meaningful step, not just at delivery - *Ask early, not late:* if anything about the goal is ambiguous, surface it before implementation, not after - *Validate direction before depth:* confirm the approach is right before building it out fully - *Checkpoint messages should include:* what was just completed, what comes next, and any open questions ### Promotion to Stable After a draft skill has been used successfully on at least one task with user confirmation: 1. Remove `status: draft` and `priority: high` from frontmatter 2. Add `validated: <date>` and a note on what the validation task was 3. Run a full distillation pass to incorporate any corrections from the task 4. Update the orchestrator's feature→skill map if one exists ### Promotion Gate for User Corrections When the trigger is a user correction, promotion requires that the corrected guidance be captured in the *canonical existing skill* before you treat the lesson as stable. Gate checklist: 1. **Patch the owning skill, not just memory:** update the skill that future agents will actually consult for that topic. 2. **Rewrite the rule in process terms:** convert "the user said X was wrong" into a reusable decision rule or anti-pattern. 3. **State the relaunch boundary explicitly:** if the fix depends on parent-shell state (for example `LD_LIBRARY_PATH`), document that runtime changes inside an already-running process do not repair the issue. 4. **Only then promote:** do not count the lesson as distilled until the canonical skill reflects the correction. Example: ROS 2 / `LD_LIBRARY_PATH` corrections belong in `skills/isaac-sim-ros2-bridge/SKILL.md`, framed as a launch-time environment rule, not as a one-off note in task memory. ### Anti-patterns for draft skills - Running a draft skill to completion without checking in. Direction may be wrong. - Skipping the user notice. They need to know confidence is lower. - Promoting to stable after one pass with no user feedback.