active-learning

name	active-learning
description	Use when building an active learning loop for low-label predictive tasks, where a model is trained on a small labeled set, used to score an unlabeled pool, and then used to select the next batch for annotation or review.

Active Learning

Overview

Use this skill when the goal is to improve a classifier or similar predictor with very limited labeled data by iterating between training, scoring, selecting examples, and annotating new batches.

This skill is for agent-assisted active learning workflows, not one-off supervised training.

Start with a short interview before changing code. Ask only the minimum needed to determine:

• how to pull the data
• whether the data is labeled, partially labeled, or unlabeled
• what the task type is:
  • classification
  • multilabel classification
  • ranking
  • extraction
  • another predictive task
• what the prediction unit is:
  • row
  • document
  • sentence
  • chunk
  • message
• what the label schema is
• what metric matters most
• whether there is a held-out validation or test set
• what the annotation budget is:
  • examples
  • rounds
  • time
  • money
• whether LLM-as-judge is allowed for weak evaluation or triage

Defaults

Use these defaults unless the user overrides them:

• use existing labeled data if available
• if labels are missing or too small, create a small seed set first
• use Argilla as the default annotation tool
• start with a simple baseline model before adding more complex selection logic
• start with uncertainty-based sampling for standard single-label classification
• add diversity filtering when duplicates or near-duplicates are likely
• evaluate after every annotation round
• keep final evaluation separate from the active learning pool
• use bounded runs rather than open-ended loops

Loop

A standard active learning loop should look like this:

1. build or confirm the initial labeled set
2. train a baseline model
3. score the unlabeled pool
4. select the next annotation batch using a defined heuristic
5. send the batch for annotation
6. merge newly labeled examples back into the labeled set
7. retrain and re-evaluate
8. repeat until stopping criteria are met

Always log what changed between rounds.

Seed Set

When labeled data is very small:

• use existing labels if they exist
• otherwise create a small seed set first
• prefer a diverse seed set over a purely random one when possible
• avoid a seed set made entirely of majority-class examples when class imbalance is known

Sampling Strategies

Choose and log the selection strategy explicitly.

Common strategies include:

• uncertainty sampling
• margin sampling
• entropy sampling
• diversity-aware sampling
• class-balance-aware sampling
• hybrid sampling:
  • uncertainty + diversity
  • uncertainty + heuristics
  • uncertainty + LLM triage

For standard single-label classification, uncertainty sampling is a good default.

Be careful:

• pure lowest-confidence sampling can over-select duplicates
• pure uncertainty can over-select outliers or noisy examples
• add deduplication or diversity controls when possible

Annotation

Use Argilla by default when no annotation tool is specified.

When setting up annotation:

• define the dataset fields clearly
• define the label schema clearly
• preserve review metadata:
  • model prediction
  • confidence or uncertainty score
  • selection reason
  • iteration number
• keep annotation provenance by round

If the user already has another annotation system, adapt to that system instead of forcing Argilla.

Evaluation

Prefer human-labeled validation for actual model evaluation.

Track at least:

• primary metric
• labeled set size
• unlabeled pool size
• selected batch size
• iteration number

If possible, keep a stable held-out validation set that is never reused as annotation input.

Do not report pool-selection performance as final model quality.

LLM as Judge

A stronger LLM may be used as a judge when the user allows it.

Possible uses:

• weak interim evaluation when gold labels are unavailable
• triage for borderline examples
• disagreement analysis
• ranking or scoring candidate predictions
• adjudication support

Do not treat LLM judgment as default ground truth unless the user explicitly wants that.

If using LLM-as-judge, log:

• judge model
• judge prompt or rubric version
• whether the judge output is:
  • label
  • score
  • ranking
  • rationale
• whether it is used for:
  • evaluation
  • prioritization
  • adjudication
  • pseudo-labeling

Keep human-gold metrics and judge-assisted metrics clearly separate.

Stopping Criteria

Define stopping criteria before running many rounds.

Examples:

• fixed number of rounds
• fixed annotation budget
• validation metric plateau
• annotator capacity limit
• time or cost cap

Default to a bounded run if the user has not specified a stopping rule.

Outputs and Logging

Keep structured outputs for every round.

At minimum log:

• iteration number
• model or program version
• labeled set size
• unlabeled pool size
• selected batch size
• sampling strategy
• evaluation metric
• annotation destination
• notes on observed failure modes

Useful files may include:

• results.tsv
• selection_round_XX.jsonl
• annotation_round_XX.jsonl
• notes.md

Use Weights & Biases when the project already uses it or the user wants centralized tracking.

Failure Modes

Watch for:

• duplicate-heavy candidate pools
• uncertain but low-value outliers
• class imbalance dominating selection
• label schema confusion
• evaluation leakage
• using LLM-judged metrics as if they were human-gold metrics
• repeatedly selecting near-identical examples

If performance is not improving, inspect:

• the selected examples
• annotation quality
• class distribution
• whether the selection heuristic is exploring useful boundaries