| name | house-trainer |
| description | Operate the EP-T house-method QLoRA trainer end-to-end on the T480 eGPU — dataset builds (T2), training/eval/GGUF (T3+), verification, and the data-quality levers. Use when training, evaluating, rebuilding the dataset, verifying a T-task, or extending Workstream T. |
house-trainer — the EP-T QLoRA pipeline playbook
Everything here was proven live on the trainer (T0–T3, LOG 2026-07-22..24). Follow it and
you skip a week of Windows-ROCm debugging. Read the two absolutes first.
Absolutes
- Privacy: job-application/CV content NEVER enters training data. Every build runs
the leak gate (
scripts/train/lib/privacy.mjs → scripts/longview/lib/leak_gate.mjs).
Quarantined sids (live home + sessions workspace quarantine.json) are dropped
structurally. Generated rows are git-ignored — never commit them. Never weaken
scripts/train/dataset/personal_terms.json to make a build pass.
- Honesty: rubric (
scripts/train/eval/rubric.md) is frozen BEFORE training. If a
run invalidates an earlier number, the correction goes on the record
(docs/train/T3-eval-report.md v1→v2 precedent). Tuned-worse is a valid, logged result.
Liveness — NEVER trust a tqdm log for a long GPU job
The eGPU wedges transiently (TB3/RDNA4): a step hangs, the process stays alive but blocked,
working set collapses to ~MB, and the tqdm log freezes looking exactly like "slow". A DPO run
sat wedged at step 8/38 for 6 h this way. Prove liveness, don't infer it:
- log mtime vs now (
stat -c %y <log>) — hours stale on a minutes-long job = wedged;
- two CPU-time snapshots ~90 s apart (
Get-CimInstance Win32_Process ... UserModeTime+KernelModeTime)
— a live job burns CPU seconds, a wedged one doesn't;
- advancing artifacts (trainer/checkpoint-*, *_result.json, adapter files).
Recover:
Stop-Process -Id <pid> -Force (PowerShell; use $procId, $id: is a parse error) →
device-matmul health check → relaunch with a lower max_length (memory pressure wedges gfx1200).
Environment (measured, don't rediscover)
- GPU: RX 9060 XT gfx1200, 15.92 GiB, Razer Core X TB3, native-Windows ROCm.
- Trainer venv python:
C:\Users\nsdha\.unsloth\studio\unsloth_studio\Scripts\python.exe
(torch 2.11.0+rocm7.13.0 / hip 7.13.99004 / Unsloth 2026.7.4 / bnb rocm714).
- Training requires a vcvars64 shell (Triton JIT): write a CRLF .bat that
calls
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\Build\vcvars64.bat
then runs the venv python; launch from Git Bash as cmd.exe //c '<single-quoted-path>'
(double slash beats MSYS path-mangling; never pipe the launch through head).
Eager eval (run_eval.py) needs no vcvars.
- llama.cpp ROCm build:
~\.unsloth\llama.cpp\build\bin\Release\ (llama-server.exe,
llama-quantize.exe); bundled gguf-py for structural reads. node: ~\.unsloth\node.
- Parallelism 1: one GPU job at a time, batch 1 (+grad accum), workers 0.
- Disk: stage fp16 downloads + merges on D: (
HF_HOME=D:\t3-merge\hf,
T3_GGUF_DIR=D:\t3-merge\gguf) — ~30 GB peak for a 7B; C: cannot take it.
- Corpus: home clone
D:\benny-home (PRIME_SILO_HOME), memo-ray ~\.mem0ray\data
(80,555 entities), big LONGVIEW workspace
D:\benny-home\benny\workspaces\sessions_v1 (376 JSON cards + data_out prose).
Pipeline (each step gated)
T2_TRACE_MAX_ENTITIES=80555 T2_TRACE_MAX_ROWS=5000 node scripts/gates/t2.mjs
node --test scripts/train/tests/build_dataset_test.mjs scripts/train/tests/build_dataset_v3_test.mjs
node scripts/train/audit_sample.mjs --n 200
<venv-python> scripts/train/eval/run_eval.py --mode base --out scripts/train/eval/out/base.json
T3_EPOCHS=2 T3_A_OVERSAMPLE=2 <venv-python> scripts/train/qlora/train_qlora.py
<venv-python> scripts/train/eval/run_eval.py --mode tuned --adapter scripts/train/qlora/out/adapter --out scripts/train/eval/out/tuned.json
HF_HOME=D:\t3-merge\hf T3_GGUF_DIR=D:\t3-merge\gguf <venv-python> scripts/train/qlora/merge_gguf.py
python scripts/gates/t3.py
Timings: NLL-only eval (--no-gen-match) ~15 min; full eval with greedy tool-match ~1 h;
train ~100 min; merge ~1–1.5 h (dominated by fp16 download if not cached).
Architecture invariants (why it looks the way it does)
scripts/train/qlora/format.py is the ONE formatting module — training labels and eval
NLL both come from encode_nll (prompt masked with -100). Never let them diverge, or
the base-vs-tuned delta measures prompt drift, not learning.- Training uses plain HF Trainer over pre-tokenized rows, NOT Unsloth's SFTTrainer
dataset path — its multiprocess tokenize-map dies on Windows (spawn/dill can't import
the generated UnslothSFTTrainer;
dataset_num_proc=1 doesn't help).
- Eager paths import
torchao_stub.py before transformers (no Windows-ROCm torchao).
- Unsloth writes GGUFs to
<requested-dir>_gguf — glob both.
- RAG stays DISABLED in eval — the number must be the fine-tune's own contribution
(method in weights, facts in RAG; no KG fact-recall rows, no synthetic paraphrases).
Data levers & current state
docs/train/DATA-PLAN-v3.md is the live plan; docs/train/T2-dataset-card.md has the
v1→v2 defect history (the 63%-empty-args lesson: audit targets, not just schemas —
median target length and duplicate-rate are the tells). Stream A sources are in
scripts/train/lib/{corpus_v3,streams_v3}.mjs; add a new source = new reader + pair
builder + red-first test + schema source.type + leak-gate wiring in build_dataset.mjs.
Rollups/KG = RAG material, never training rows.
T4 — serving behind the router (LM Studio + eGPU)
Additive candidate: runtime/benny/router/tuned_engine.py registers house/… in MODEL_REGISTRY
- wraps
resolve_executor (default engine unchanged, reversible). Serve the GGUF via LM Studio
on the eGPU (copy into ~/.lmstudio/models/<pub>/<repo>-GGUF/, lms load <modelKey> --gpu max --identifier <id>; lms import hangs headless). Gate scripts/gates/t4.py = structural
(litellm-free) + live offload through the ADR-004 run_task. Gotchas: LM Studio 400s
response_format:json_object (fixed provider-agnostically in offload/gate.py::run_judge); use a
FAST non-reasoning judge (gemma-3-4b), different family from the executor (anti-collusion); never
run the live gate during a training run. See [[lmstudio-egpu-serving]].
T5 — DPO (preference tuning)
Self-generated hard negatives: scripts/train/dpo/build_prefs.py asks the SERVED SFT model
(LM Studio) for greedy answers on TRAIN-split prompts; where wrong vs the corpus reference, emit
(chosen=reference, rejected=SFT-answer). Filter to method signal — keep tool-selection
errors + voice-drift; DROP arg-value mismatches (that's memorising RAG facts, against the doctrine).
train_dpo.py: PatchDPOTrainer() first, load the SFT adapter as policy, ref_model=None,
beta 0.1, lr 5e-6 (≪ SFT's 2e-4), 1 epoch, max_length ≤1024. SFT baseline eval = reuse the last
SFT tuned.json (same adapter+split, deterministic). Gate scripts/gates/t5.py: GREEN iff
dpo.agg_nll ≤ sft.agg_nll + merged GGUF loads. DPO-not-beating-SFT is a valid logged result.
Verification protocol (author≠verifier)
Log as claude-tN-verifier. Reproduce, don't trust: re-run the NLL instrument fresh
(deterministic — must match to ~6 decimals), own split-overlap check, NEGATIVE tests
(swap base/tuned reports → expect RED tuned_worse; empty GGUF dir → RED gguf_missing;
seeded CV row → t2 RED), own llama-server /health smoke, then gate re-run. Move
VERIFY→DONE on the board + LOG with caveats stated (same-session verify is a named
caveat, precedent T2/T3).
Current board state (2026-07-25)
T0/T1/T2/T3 DONE+verified. T3 v3 (data-plan-v3, all pre-registered criteria passed):
tuned Qwen2.5-Coder-7B −57% agg NLL vs base (2.6195→1.1253, A −38.3%, RAG off);
GGUF D:\t3-merge\gguf_gguf\…Q4_K_M.gguf, also in LM Studio as qwen2.5-coder-tuned.
T4 DONE (in VERIFY) — tuned wired behind the router, live ADR-004 offload passed, judge 1.0.
T5 (DPO) in progress — 303 method pairs, first run wedged (relaunched, max_len 1024).
Owner still owes the 200-row gold hand-audit (human-signed). KR1.5 closes on T5 verify.
