few-shot-curator

name	few-shot-curator
description	When a task needs a specific format or style and zero-shot keeps drifting, give 2-5 carefully chosen examples instead of more instructions. Use for extraction/classification/formatting/tone tasks the model keeps getting subtly wrong. Trigger with /few-shot-curator or "add examples", "show it a few examples", "few-shot this".
version	0.1.0
user-invocable	true
metadata	{"emoji":"🎯"}

few-shot-curator

When zero-shot drifts on format or style, a few well-chosen examples steer it better than another paragraph of instructions. The trick is WHICH examples: relevant, diverse, correct.

Why this exists (evidence)

• In-context learning is strongly sensitive to which examples you pick: good example selection beats random examples, and relevant + diverse examples beat more-but-noisy ones (Liu et al., "What Makes Good In-Context Examples for GPT-3?", and the broader ICL literature). The win is from curation, not quantity.
• It targets the failure where the model "understands" the task but keeps producing the wrong shape/tone, and adding more prose instructions does not fix it.

When to use

• Output format/structure the model keeps getting subtly wrong (a specific JSON, a commit-message style, a tone).
• Extraction/classification where edge cases need to be shown, not described.
• NOT when zero-shot already nails it (examples just cost tokens), and not as a crutch for an unclear task (fix the instruction first).

The method

1. Pick 2-5 examples, curated:
   • Relevant: close to the actual inputs you expect.
   • Diverse: cover the range and the edge cases (the tricky ones, not five near-duplicates).
   • Correct: each example's output is exactly what you want (a wrong example teaches the wrong thing).
   • Format-faithful: examples show the EXACT output shape you want back.
2. Order intentionally: put the most representative/important last (recency in the prompt carries weight).
3. Keep them lean: shortest examples that still demonstrate the pattern (compose with prompt-compression).
4. Iterate: if it still drifts on a case, add an example of THAT case rather than more prose.

How to run it

• Maintain a small bank of golden input->output pairs for recurring task types; select the relevant few per task.
• For classification, include at least one example per class and per known confusable.
• Verify the examples are correct before using them (a bad few-shot example is worse than none).

Composes with

• structured-output: examples + a schema = strong, consistent shape.
• prompt-compression: keep the examples minimal so they do not bloat context.
• self-consistency: if unsure which examples help, test a couple of sets and keep the one that yields consistent correct output.

Honest limits

• Examples cost context tokens every call; do not few-shot what zero-shot already does.
• Wrong/biased examples actively teach the wrong behavior; curate, do not grab the first samples.
• More examples is not better past a few; relevance and correctness beat count.