tap-model-training

## What this skill is for `muvon/octomind-embed` is the bi-encoder that decides which capability auto-activates from a user's prompt. It is fine-tuned from BGE-small-en-v1.5 on the triggers authored in `capabilities/*/config.toml` and the `semantic(phrase)` rules in `skills/*/SKILL.md`. Whenever those source files change, the model is stale. This skill walks through the retrain-and-publish loop and the recovery paths. The cross-encoder reranker (`muvon/octomind-rerank`) is **intentionally not maintained**. Production validation showed it doesn't help at the current capability count (~75 labels); the bi-encoder + a well-tuned runtime gate is the production path. ## When to retrain You MUST retrain (and republish) when the supervision set changes: | Change | Retrain? | | ------------------------------------------------------------ | -------- | | Added `capabilities/<new>/config.toml` | yes | | Removed a capability | yes | | Edited `triggers = [...]` in any `config.toml` | yes | | Added or edited `semantic(...)` rules in any `SKILL.md` | yes | | Edited `_oos` curated phrases in `build_dataset.py` | yes | | Agent/playbook/docs change with no trigger touched | no | | Code change in `octomind/` runtime only | no | | Threshold/margin tweak in `capability.rs` / `skill.rs` | no (just rebuild octomind) | ## Standard retrain flow ALWAYS run inside `tmux` or `nohup` — sessions get disconnected. ```bash cd model uv sync # if env not warm tmux new -s train ANTHROPIC_API_KEY=sk-ant-... bin/train --llm --resume --iterations 2 # Ctrl+B then D to detach. Reattach with: tmux attach -t train ``` What runs: 1. `augment_llm.py --resume` — paraphrases ONLY the new `(label, trigger)` pairs against `data/intents.jsonl`. Skips already-augmented ones. Cost: ~$0.10–$2 depending on how many new triggers. Time: 3–5 min. 2. `build_dataset.py` — rebuilds `pairs.jsonl`, `triplets.jsonl`, `holdout.jsonl` from `capabilities/` + `skills/` + LLM intents + OOS sink + multi-turn surfaces + realistic typo augmentation. 3. Train iter 1 with CachedMNRL + GISTEmbed + Matryoshka. ~3-6h on GPU, 6-12h on CPU. 4. Re-mine triplets with iter-1 model (retrieval-aligned + NV-Retriever positive-aware filter). 5. Train iter 2. Same time as iter 1. 6. `eval.py` against holdout (in-distribution). 7. `eval_gate.py` against `data/eval_real.jsonl` (real-user distribution — informational at this step, blocks at publish). 8. `export_onnx.py` — produces `checkpoints/embed-<ts>/onnx/` for ORT-based consumers. When it finishes, publish: ```bash HF_TOKEN=hf_... bin/publish ``` `bin/publish` runs `eval_gate.py` against `eval_baselines.json` first and aborts on regression (top1_acc drops >1pt, null_fpr rises >1pt, any per-cap recall drops >5pts). Use `--force` only for documented emergencies. After production rollout, record the new baseline so the NEXT publish gates against it: ```bash uv run python scripts/eval_gate.py --model muvon/octomind-embed \ --write-baseline eval_baselines.json ``` ## Resuming an interrupted training SentenceTransformerTrainer auto-checkpoints at the end of each epoch (`save_strategy="epoch"`, `save_total_limit=2`). Recovery: ```bash # 1. check what got saved ls checkpoints/embed-*/checkpoint-* ``` If you see `checkpoint-N/` dirs, there's state. Resume: ```bash uv run python scripts/train.py --resume ``` That picks the latest `checkpoints/embed-*/` dir and continues from the highest-numbered `checkpoint-N`. Use `--resume-from <path>` for an explicit run dir. Mid-iter-1 interruption — finish iter 1, then continue iter 2 manually: ```bash # 1. finish iter 1 uv run python scripts/train.py --resume # 2. re-mine triplets with iter-1 model ITER1=$(ls -td checkpoints/embed-*/ | head -1 | sed 's:/$::') uv run python scripts/build_dataset.py --neg-embed-model "$ITER1" # 3. train iter 2 fresh uv run python scripts/train.py # 4. eval + export ITER2=$(ls -td checkpoints/embed-*/ | head -1 | sed 's:/$::') uv run python scripts/eval.py --run "$ITER2" uv run python scripts/eval_gate.py --model "$ITER2" uv run python scripts/export_onnx.py --run "$ITER2" ``` `--resume` works only for the modern `train.py` path (CachedMNRL + GIST + Matryoshka). `--legacy` has no checkpoint-resume. ## Calibrating the runtime gate After every retrain, recalibrate the runtime threshold/margin against eval_real: ```bash RUN=$(ls -td model/checkpoints/embed-*/ | head -1 | sed 's:/$::') uv run python model/scripts/calibrate_thresholds.py --model "$RUN" --target-fpr 0.05 ``` It sweeps τ × δ over `data/eval_real.jsonl`, prints the Pareto front, and recommends the operating point with highest gate_acc whose null_fpr ≤ target. Copy-paste the recommended constants into: - `octomind/src/mcp/core/capability.rs:772` → `AUTO_ACTIVATE_THRESHOLD` - `octomind/src/mcp/core/capability.rs:781` → `AUTO_ACTIVATE_MARGIN` - `octomind/src/mcp/core/skill.rs:66` → `SEMANTIC_DEFAULT_THRESHOLD` - `octomind/src/mcp/core/skill.rs:78` → `SEMANTIC_MARGIN` The threshold often drops after fine-tuning (0.55 → ~0.45) because the FT model puts every matched positive well above the floor; the margin becomes the binding constraint, not the floor. ## Common mistakes 1. Using `--skip-build` when triggers changed. The build is what converts new triggers into training surfaces. `--skip-build` reuses the previous `pairs.jsonl`, so new triggers don't show up in the training set, and the resulting checkpoint inherits the old weaknesses. The new run is wasted compute. Always rebuild after a catalog change. 2. Forgetting `--llm` after adding triggers. `--llm` is what kicks off `augment_llm.py` to generate user-style paraphrases of the new triggers. Without it, the new triggers only get rule-template expansion, which is much narrower. Use `--llm --resume` to skip already-paraphrased triggers. 3. Running outside tmux/nohup. Long training (5h+) over SSH dies when the connection drops. Always wrap in tmux or nohup. 4. Skipping the publish gate. `bin/publish --force` exists for emergencies. Don't use it as habit — silent regressions ship. 5. Not recording the new baseline after promotion. If you don't run `eval_gate.py --write-baseline` after a successful publish, the next publish has nothing to gate against and any future regression slips through. 6. Editing trigger lists without thinking about overlap. When two capabilities share vocabulary (e.g. all `messaging-*`, all `legal-*`, `webfetch` vs `scraping`), the model gets confused. Use PLATFORM-specific or JURISDICTION-specific vocabulary in each cap's triggers — not the same generic verbs across the family. ## Key files - `model/bin/train` — full pipeline entrypoint. - `model/bin/publish` — gate-protected HF upload. - `model/configs/default.yaml` — loss stack, batch sizes, eval paths. - `model/scripts/build_dataset.py` — surface generation, OOS sink, multi-turn, typo aug, hard-negative mining. - `model/scripts/train.py` — modern trainer (CachedMNRL + GIST + Matryoshka) + `--resume` flag. - `model/scripts/eval_gate.py` — publish-time regression gate. - `model/scripts/calibrate_thresholds.py` — runtime τ/δ sweep. - `model/scripts/build_eval_seed.py` — frozen real-user eval set generator. Hand-edit `data/eval_real.jsonl` to add production-observed failure modes. - `model/eval_baselines.json` — recorded baseline; the publish gate reads this. - `model/data/intents.jsonl` — LLM paraphrases. `augment_llm.py --resume` skips entries already here. ## When NOT to retrain If the symptom is "auto-activation picks the wrong capability for X but the triggers are correct", retraining usually won't fix it. Three other levers first: 1. Run `calibrate_thresholds.py` to see if the current τ/δ in the Rust constants matches the FT model's distribution. 2. Add the failing prompt to `data/eval_real.jsonl` with the correct label, then re-run `eval_gate.py` to confirm it's an actual model problem and not an eval-distribution gap. 3. Check the trigger lists for the competing capabilities — if two neighbors share generic vocabulary, the fix is differentiating triggers, not retraining alone. Retrain when the triggers themselves change, not when production behavior is unexpected.