Procedures for competing in LLM-era ML competitions — NLP/text classification, LLM fine-tuning tracks, and prompt/agent contests. Use when the task is a text or LLM competition graded by a hidden metric under latency/memory caps: rebuild the metric locally first, climb the prompt→few-shot→LoRA→full-finetune efficiency ladder only as far as the eval demands, audit label/data quality before touching architecture, and report mean of 3 seeds. Use when asked to improve an NLP leaderboard score, fine-tune a model for a contest, engineer a prompt against a held-out set, or check a text dataset for contamination/leakage.
Procedures for competing in LLM-era ML competitions — NLP/text classification, LLM fine-tuning tracks, and prompt/agent contests. Use when the task is a text or LLM competition graded by a hidden metric under latency/memory caps: rebuild the metric locally first, climb the prompt→few-shot→LoRA→full-finetune efficiency ladder only as far as the eval demands, audit label/data quality before touching architecture, and report mean of 3 seeds. Use when asked to improve an NLP leaderboard score, fine-tune a model for a contest, engineer a prompt against a held-out set, or check a text dataset for contamination/leakage.
Procedures for running an LLM-era ML competition the way strong competitors do: an
NLP/text contest, an LLM fine-tuning track, or a prompt/agent contest. This is the
text-and-LLM specialization of general competition practice; the broad six-phase backbone
(detect metric → plan → fan out → adversarially verify → evaluate-gate → ship) is inherited
from the skill. Read that first if the task has a /
shape; this skill fills the LLM-specific content: the eval rebuild, the efficiency ladder, the
data-centric loop, inference-budget realism, and the failure-mode list.
ml-competition
train.csv
sample_submission
Scope and honesty. This skill has a deliberately thin evidence base. Where a claim is
well-established public practice it is stated plainly; where a choice is contested or
context-dependent it is marked as such. Do not overclaim. When a number would help, prefer
"measure it on your held-out set" over a borrowed constant.
Operating principles
The hidden metric is the only judge; your local rebuild of it is your ground truth. A
leaderboard scores a fraction of a hidden set and is the largest source of self-deception.
Trust a faithful local eval; treat each public-LB move as a weak signal until the local
eval agrees.
Climb the efficiency ladder only as far as the eval demands. Prompt → few-shot →
LoRA/QLoRA → full fine-tune. Each rung costs more and overfits more easily; stop at the
cheapest rung that clears the bar.
Data quality beats architecture changes. A label audit, a deduplication pass, or a
contamination check usually moves the metric more than swapping a model. Spend there first.
Grade under the real budget. Most LLM contests cap latency and/or memory. A solution
that wins offline but violates the cap scores zero. Measure latency/memory from the start.
Report mean of ≥3 seeds. LLM fine-tuning and few-shot results have real seed variance;
a single run is not a result.
Phase A — EVAL-FIRST: rebuild the competition metric locally
Do this before touching a model. A model you cannot score faithfully is a model you are
tuning blind.
Pin the exact metric and its direction. Macro-F1 vs micro-F1, exact-match vs token-F1,
ROUGE-L vs ROUGE-1, pass@1 vs pass@k, BLEU variant, a custom weighted score — each changes
the whole strategy. Read the rules; if ambiguous, dispatch the research skill for the
public metric definition. Confirm whether higher or lower wins.
Re-implement the scorer and unit-test it. Reproduce the official scorer (or wrap the
official script) and verify it on any provided examples until your numbers match the
public baseline to the decimal. If they do not match, you have the wrong metric — stop and
fix it.
Carve a held-out set the model never sees. Split a portion of labelled data the model
(and, for prompt contests, the prompt-author) never reads. For prompt/agent contests
this is non-negotiable: prompts iterated against examples you keep staring at silently
memorize them. Keep a blind slice you only score on, never inspect.
Lock the validation scheme to the task structure. Stratify by label for imbalanced
classification; group by document/author/source when rows share an entity (GroupKFold) so
the same entity never spans train and held-out; respect time order for temporal text.
Establish baselines. A trivial baseline (majority class / copy-input / empty answer)
and one cheap real baseline (a zero-shot prompt, or a small fine-tuned encoder like a BERT
family model for classification) set the floor and validate the harness end-to-end.
Exit: the local scorer reproduces the public baseline, a blind held-out set exists, the
validation scheme matches the data structure, and a baseline number is recorded.
Phase B — THE EFFICIENCY LADDER: stop at the cheapest rung that clears the bar
Climb in order; promote to the next rung only when the current rung plateaus below target on
your held-out set. Each rung is more expensive and more prone to overfitting.
Prompt engineering (rung 0). Cheapest, no training. Write a clear instruction with an
explicit output schema; constrain the output format so it is machine-scorable. Add
reasoning steps only when the task is genuinely multi-step — chain-of-thought helps
arithmetic/multi-hop reasoning but adds latency without accuracy gain on plain
classification (Wei et al. 2022, arXiv:2201.11903). For format guarantees use constrained
decoding (outlines/guidance) or structured-output APIs.
Few-shot / in-context (rung 1). Add 3–8 labelled exemplars. Choose edge cases, not
prototypes; include one "tricky" example per class. Balance classes. Few-shot frequently
closes most of the gap to a small fine-tune at zero training cost. Treat example selection
and ordering as hyperparameters and score them on the held-out set.
LoRA / QLoRA fine-tune (rung 2). When prompting/few-shot plateaus below target and you
have a few hundred-plus clean labelled examples, parameter-efficient fine-tuning is usually
enough. LoRA freezes the base weights and trains small low-rank adapters (Hu et al. 2021,
arXiv:2106.09685). QLoRA backs the frozen base with 4-bit NF4 quantization so a single
consumer/prosumer GPU can fine-tune a large model, with quality reported comparable to
16-bit fine-tuning on the studied setups (Dettmers et al. 2023, arXiv:2305.14314). Start
with a modest adapter rank, train for a small number of epochs, and watch the held-out
metric for overfitting (it turns up fast). Use existing skills (peft, trl-fine-tuning,
unsloth, axolotl, bitsandbytes) — do not reimplement training.
Full fine-tune (rung 3). Reach for this only when LoRA/QLoRA is demonstrably
capacity-bound (it has plateaued and the gap is large) and you have the compute,
abundant clean data, and a budget cap that the larger result still satisfies. It is the
most expensive rung and the easiest to overfit; many text competitions are won without it.
Decision rule: promote a rung only on held-out evidence that the current rung has
plateaued below target — never because the next rung sounds stronger. Record the held-out
score at each rung so the promotion is auditable.
Phase C — DATA-CENTRIC ITERATION: audit data before architecture
Spend the first real iteration on the data, not the model.
Label-quality audit. Sample mislabelled-looking rows (e.g. confident model errors,
high train loss, annotator-disagreement rows) and re-check them by hand. Fixing label
noise is often the single highest-impact lever; it beats most architecture swaps. This is
the data-centric view popularized in applied practice (Thakur, Approaching (Almost) Any
Machine Learning Problem; see references).
Deduplication. Remove exact and near-duplicate texts within the training set; duplicates
inflate apparent performance and waste capacity. Large-corpus work shows pervasive
near-duplication and its harms (Dodge et al. 2021, arXiv:2104.08758). Use MinHash/LSH or
embedding-cosine thresholds for near-duplicates.
Contamination / leakage check against the test set. Verify no test (or held-out)
example, or a near-duplicate of one, appears in training data — and, for prompts/agents,
that no answer is encoded in an id, filename, or timestamp. Benchmark contamination is a
documented, easy-to-miss failure that silently inflates scores (Sainz et al. 2023,
arXiv:2310.20410; Deng et al. 2024, arXiv:2311.04850). Run n-gram/substring overlap and
embedding-nearest-neighbour checks between train and the hidden/held-out split where you
have access to both.
Targeted augmentation, measured. Add data only where the label audit shows a real gap
(an under-represented class, a hard slice). Re-score on the blind held-out set; keep
the change only if it moves that number.
Exit: labels audited on the error-prone slices, duplicates removed, a contamination/leakage
check has run and is clean (or the leak is fixed), and any augmentation is justified by a
held-out gain.
Phase D — INFERENCE-BUDGET REALISM: grade under the cap
LLM contests routinely cap latency, memory, or both. A solution that violates the cap scores
zero regardless of accuracy.
Read the cap first — max latency per example, GPU/CPU and memory, max model size,
offline-vs-online, token limits. Treat it as a hard constraint on rung and model size.
Measure your real footprint — wall-clock latency per example and peak memory on the
target-class hardware, not a guess. A few-shot prompt with a long context can blow a
latency cap; a full-precision large model can blow a memory cap.
Quantize to fit, then re-score. 8-bit and 4-bit (NF4) quantization cut memory ~2–4×
with typically small quality loss, and QLoRA shows 4-bit bases can even be fine-tuned
(Dettmers et al. 2023, arXiv:2305.14314). Post-training 4-bit schemes (awq, gptq,
bitsandbytes, gguf for CPU) are the usual tools. Quantization is not free — always
re-score the quantized model on the held-out set; the loss is task-dependent, so measure it
rather than assume it.
Right-size the model. A smaller model that fits the budget and is fine-tuned on clean
data often beats a larger model you must cripple to fit. Pick the largest model that clears
the cap with margin, not the largest model that exists.
Exit: the chosen solution meets the latency/memory cap with margin on target-class hardware,
and its held-out score is the post-quantization score.
Phase E — FAILURE MODES TO HUNT (reject-mode gate)
Run these as an adversarial pass before trusting any score (the LLM analog of a leakage gate).
Overfitting the public leaderboard. Tuning toward the public split until private-LB
collapse. Mitigation: trust local held-out CV; track the public-LB↔local gap; a widening
gap means you are chasing the public split. Do not burn submissions probing.
Prompt leakage / blind-set contamination. Iterating a prompt against examples you keep
reading silently memorizes them; the gain does not transfer. Mitigation: a blind held-out
slice scored only, never inspected (Phase A.3).
Train/test (benchmark) contamination. Test text leaking into training or pretraining
inflates scores (Phase C.3). Mitigation: run the overlap + nearest-neighbour checks.
Seed variance reported as signal. A one-run improvement inside the noise band.
Mitigation: report mean ± std over ≥3 seeds; fine-tuning is genuinely seed-sensitive
(Dodge et al. 2020, arXiv:2002.06305). A change inside seed std is not an improvement.
Format/parse failures. A correct answer in the wrong format scores zero. Mitigation:
constrained/structured decoding and a format validator before submission.
Latency/memory cap violation. Offline-best, online-disqualified (Phase D). Mitigation:
measure under the cap from the start.
For each finding: FIX (and show the diff / the corrected eval) or DEFEND with held-out
evidence. Resolved = rebuttal written and any fix applied.
Ship
Refit/finalize the chosen rung, produce the submission in the exact required format
(validate programmatically against the sample), and record a reproducible write-up: the metric
and its rebuild, the ladder rung that won with its held-out score, the data-centric fixes, the
seed mean ± std, the post-quantization footprint vs the cap, and the public-LB↔local gap.
Self-check before delivering
Answer each yes/no honestly; any "no" is a blocker.
Did you rebuild the competition metric locally and confirm it matches the public baseline
before training anything?
Is there a blind held-out slice that the model — and, for prompt contests, the
prompt-author — never saw?
Did you stop at the cheapest ladder rung that cleared the bar, with a recorded held-out
score justifying each promotion?
Did you run a deduplication and a train/test contamination check, and is the result clean
(or the leak fixed)?
Is the reported score a mean over ≥3 seeds (with std), and the held-out score the
post-quantization number under the real latency/memory cap?
Does the submission pass a programmatic format check against the sample?
References
Full citation list with verified URLs: references/sources.md.
