| name | knowledge-distillation-a-survey |
| description | Choose what kind of knowledge to transfer between teacher and student models: response, feature, or relational, and decide among offline, online, self, or cross-modal distillation schemes. Best for distillation strategy selection, capacity-gap diagnosis, and transfer planning. Activate on "knowledge distillation", "teacher-student", "soft labels", "feature distillation", "online distillation", or "cross-modal transfer". NOT for generic compression checklists or unrelated training work. |
| license | Apache-2.0 |
| allowed-tools | ["Read","Write","Edit","Glob","Grep"] |
| metadata | {"category":"Research & Academic","tags":["knowledge-distillation","teacher-student","transfer-learning","model-compression","cross-modal","survey"],"pairs-with":[{"skill":"knowledge-distillation-in-deep-learning","reason":"Use the application-focused companion when the design question is deployment, cascades, or distillation economics."},{"skill":"ai-engineer","reason":"System builders often need this survey lens before choosing concrete distillation infrastructure."}],"provenance":{"kind":"legacy-recovered","sourceDocument":"Knowledge Distillation: A Survey","sourceAuthors":["Jianping Gou","Baosheng Yu","Stephen J. Maybank","Dacheng Tao"],"sourceArtifact":".claude/skills/knowledge-distillation-a-survey/_book_identity.json","importedFrom":"legacy-recovery","owners":["some-claude-skills"]},"authorship":{"authors":["Jianping Gou","Baosheng Yu","Stephen J. Maybank","Dacheng Tao"],"maintainers":["some-claude-skills"]}} |
Knowledge Distillation: A Survey
Use this skill when the hard question is not "should we distill?" but "what kind of knowledge is worth transferring, through which scheme, across what capacity gap?"
When to Use
- You need to choose among response-based, feature-based, or relation-based transfer.
- A strong teacher exists, but the student keeps underperforming and you suspect a capacity-gap or representation mismatch.
- You are deciding between offline, online, self-distillation, or cross-modal transfer.
- The team wants to reuse knowledge across modalities, domains, or resolutions and needs to know what is essential versus incidental.
- You need a high-level distillation doctrine before designing the specific training stack.
NOT for Boundaries
This skill is not the primary fit for:
- Generic model compression checklists that never ask what knowledge is being transferred.
- Architecture tuning problems without a teacher-student or representation-transfer question.
- Deployment economics, cascade routing, or threshold setting once the transfer strategy is already chosen.
- Prompt-only optimization or ordinary supervised fine-tuning with no distillation mechanism.
Core Mental Models
Knowledge Has Multiple Transferable Forms
The survey's central move is to stop treating "knowledge" as one blob. Teachers can transfer:
- Response knowledge through logits, soft labels, and calibrated predictions.
- Feature knowledge through intermediate representations and abstractions.
- Relational knowledge through geometry, similarity, and structure across examples or layers.
Capacity Gap Is A Structural Constraint
Distillation fails when the student cannot represent the thing you are trying to teach. Large teacher advantage can become a liability if the transferred representation is too compressed, too entangled, or too alien for the student.
Transfer Scheme Encodes A Coordination Philosophy
- Offline distillation assumes the expert is already stable.
- Online distillation assumes peers or ensembles should co-evolve.
- Self-distillation assumes the model contains useful internal views worth aligning.
These are different assumptions about where valid knowledge lives and how it changes.
Cross-Modal Success Exposes The Essential
If the transfer survives a modality jump, the knowledge is probably abstract and portable rather than tied to surface form. Cross-modal wins are a clue about what the teacher actually knows.
