few-shot-curator

# few-shot-curator When zero-shot drifts on format or style, a few well-chosen examples steer it better than another paragraph of instructions. The trick is WHICH examples: relevant, diverse, correct. ## Why this exists (evidence) - In-context learning is strongly sensitive to which examples you pick: good example selection beats random examples, and relevant + diverse examples beat more-but-noisy ones (Liu et al., "What Makes Good In-Context Examples for GPT-3?", and the broader ICL literature). The win is from curation, not quantity. - It targets the failure where the model "understands" the task but keeps producing the wrong shape/tone, and adding more prose instructions does not fix it. ## When to use - Output format/structure the model keeps getting subtly wrong (a specific JSON, a commit-message style, a tone). - Extraction/classification where edge cases need to be shown, not described. - NOT when zero-shot already nails it (examples just cost tokens), and not as a crutch for an unclear task (fix the instruction first). ## The method 1. **Pick 2-5 examples, curated:** - **Relevant:** close to the actual inputs you expect. - **Diverse:** cover the range and the edge cases (the tricky ones, not five near-duplicates). - **Correct:** each example's output is exactly what you want (a wrong example teaches the wrong thing). - **Format-faithful:** examples show the EXACT output shape you want back. 2. **Order intentionally:** put the most representative/important last (recency in the prompt carries weight). 3. **Keep them lean:** shortest examples that still demonstrate the pattern (compose with prompt-compression). 4. **Iterate:** if it still drifts on a case, add an example of THAT case rather than more prose. ## How to run it - Maintain a small bank of golden input->output pairs for recurring task types; select the relevant few per task. - For classification, include at least one example per class and per known confusable. - Verify the examples are correct before using them (a bad few-shot example is worse than none). ## Composes with - `structured-output`: examples + a schema = strong, consistent shape. - `prompt-compression`: keep the examples minimal so they do not bloat context. - `self-consistency`: if unsure which examples help, test a couple of sets and keep the one that yields consistent correct output. ## Honest limits - Examples cost context tokens every call; do not few-shot what zero-shot already does. - Wrong/biased examples actively teach the wrong behavior; curate, do not grab the first samples. - More examples is not better past a few; relevance and correctness beat count.