eval-training-run

스타4

포크0

업데이트2026년 7월 12일 23:58

Evaluate the health of a LIVE nano training run from its Modal logs — loss trajectory vs the random floor, per-codebook convergence order, val-checkup trend and strikes, throughput stability, ETA, and infra noise triage. Use this skill when the user asks "how is training going", "is the run healthy", "evaluate the training run", "is it diverging", or wants an ETA to a step count. For evaluating a finished CHECKPOINT (sample quality, WER, overfitting) use eval-checkpoint instead; for launching/resuming runs use train-model.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

출처

diskrotrepo

diskrotrepo/nano

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Evaluate a live training run

Produces a structured health report from the run's logs. Read train-model/references/reading-logs.md for line formats and the benign-noise list; this skill is the judgement layer on top. (Hardware/throughput numbers in that doc are v8-era — the bands below supersede them.)

Two vintages of reference numbers appear below; pick by the run's codec + shape: v9-class (SpectroStream, K=24, 25 Hz, 180s segments, B200:4, EMA on) vs v8-class (DAC, K=9, 86 Hz, 60s segments, 8×H100, no EMA). If unsure, count the entries in a step line's cb[...] — 24 = v9-class.

Gather

modal app list | grep nano-train                  # find the app id + state
APP=ap-...
modal app logs $APP 2>&1 | grep -aE "step +[0-9]+/" > /tmp/steps.txt
modal app logs $APP 2>&1 | grep -aE "checkup at step|early stopping" > /tmp/checkups.txt
modal app logs $APP 2>&1 | grep -acE "Timed out .* waiting for clients|Traceback|CUDA out of memory"

(Full-log queries can hit a server-side resource limit once CLAP progress spam accumulates — always pipe through a grep filter, never page raw logs.)

modal app logs may also return only a recent tail of the history (server-side truncation — observed 2026-07-12: a run 13 checkups in returned just the last 2). Missing early checkup lines do NOT mean "no checkups ran"; reconstruct the earlier trajectory from each surviving checkup's beat previous best (X) field, which carries the prior best score.

Judge — in this order

Reference numbers: random-chance loss is ln(vocab) = ln(1024) ≈ 6.93 per codebook — vocab is 1024 for both DAC and SpectroStream, so the floor survives the codec switch (re-derive it if vocab ever changes). Train loss above 6.93 after warmup = confidently wrong, not "still learning".
Val trajectory (the verdict): checkups every 1000 steps must trend DOWN. With EMA on (v9 default, use_ema=True) the checkup score — and best.pt — are the EMA weights, not the live ones: (a) EMA smooths noise away, so consecutive worse checkups are a stronger divergence signal than they were on v8; (b) don't compare the step-line train loss (live weights) against the checkup score (EMA) as if they were the same basis. Two+ consecutive strikes while LR is still climbing through warmup = the 2026-06-12 divergence signature — stop the run, don't wait out patience. Healthy v9_stereo reference (B200:4, launched 2026-07-12): every checkup a new best through 13k — 5.2321@11k, 5.1777@12k, 5.1300@13k (just past the 10k warmup end, full lr 1.5e-4). Healthy v8 reference (bf16+qk-norm): 6.75@1k, 6.69@2k, 5.11@8k, 4.93@10k (warmup end) — every checkup a new best. Diverged v8 reference: 7.17 → 7.87 → 8.70 by step 3000. Scores are NOT comparable across vintages (different codec + K). The step line's grad field is the leading indicator: healthy is FLAT — v9_stereo sits ~0.10–0.20 at full LR; the v8 recipe ran a 0.3–2.5 band (clip=1.0 occasionally touched is fine on either). A ramp across windows precedes divergence by thousands of steps.
Per-codebook order: shallow codebooks converge first, so a checkup's cb[...] spread must be ordered low→high from cb0 (healthy v9 @13k: cb0 1.88 rising monotonically to cb23 6.46). Adjacent levels swapping in a single noisy 25-step train line is normal; judge ordering on checkup lines (averaged over 50 eval batches). cb0/the shallow levels HIGHEST or climbing = the divergence canary — cb0 led the 2026-06 explosion by ~800 steps. All K equal after thousands of steps = delay-pattern suspicion.
Throughput: reported tok/s counts global-batch × K × frames, so it is NOT comparable across codecs or shapes. v9_stereo steady state ≈ 2.95–3.05M tok/s on B200:4 (the hardwired multi-GPU config; ~3.46M tokens/step → ~0.86 steps/s, ~29s per 25-step line). v8-era: ~1.6–1.7M on 8×H100 (K=9, 60s). Isolated low windows in the first few thousand steps = late first-hit variant compiles (finite, self-quiescing). Persistently low/erratic after that = a starving rank. Ignore the step-25 line (compile-polluted average).
ETA: steps/s from the spacing of recent step lines (each line = 25 steps), then (target - current) / steps_per_s. State it with the caveat that val-driven early stopping (patience 20) may end the run first.
Infra events: count NCCL rendezvous timeouts ("waiting for clients") — one is a transient (Modal auto-retries; resume is checkpoint-safe), repeats on the same app = bad node, stop + relaunch for fresh placement. Check the benign-noise list before flagging anything else.
Checkpoint state: modal volume ls nano-ckpts <subdir>/ — best.pt updates on checkup improvements (EMA weights when EMA is on), latest.pt/step_*.pt every 5000 steps.

Report

Lead with the one-line verdict (healthy / watch / diverging / stalled / dead), then: latest checkup vs best, last train loss + cb spread, steps/s + ETA to the user's target (default: next checkpoint-compat milestone or 100k), and any infra events. Compare against the references above so "healthy" is a claim about THIS model's known-good trajectory, not vibes — and say which vintage (v9/v8) the comparison used.

이 저장소의 다른 Skills

같은 저장소

serve-model

diskrotrepo/nano

Serve nano inference locally or on Modal and exercise the generation endpoints. Use this skill when the user wants to run the server, generate or extend audio from a checkpoint, cover a hummed melody, deploy the inference API, or test /generate /extend /cover with text, lyrics, melody, or style conditioning.

2026-08-024

add-songs

diskrotrepo/nano

Add MP3s to the nano training corpus and run them through the full data-prep pipeline (upload, prepare, tokenize, optional melody/auto-tag/transcribe/ structure/key-detect, phonemize, pack). Use this skill when the user wants to add training data, ingest songs, build or grow the corpus, prepare data for training, run tokenize/melody/pack/tag/transcribe/structure/phonemize, or asks how to get their MP3s into the model.

2026-07-124

eval-checkpoint

diskrotrepo/nano

Evaluate a nano checkpoint and pick the right diagnostic for the question. Use this skill when the user wants to evaluate a checkpoint, score sample quality (CLAP / spectral / silence / onset), check overfitting (train vs val), decide whether lyric conditioning is worth the compute, sweep sampling parameters, check genre coverage, or sanity-check DAC codec fidelity.

2026-07-124

eval-lyrics

diskrotrepo/nano

Audit and evaluate nano's lyric data and lyric conditioning. Use this skill when the user wants to analyze the lyric dataset, check transcribe/phonemize coverage or backfill progress, measure the Whisper-hallucination rate, see the vocal-gender or word-count distribution, decide whether the lyric filter should run, or evaluate whether a checkpoint actually sings the words.

2026-07-124

eval-training-data

diskrotrepo/nano

Audit the health of nano's training data before a run — corpus size vs the scale rails, per-stream conditioning coverage (tags / structure / keys / lyrics / phonemes / melody), song-duration distribution, and tag/genre spread. Use this skill when the user wants to check whether the data is ready to train, see how many songs have each conditioning stream, check melody or phoneme coverage, audit corpus size, or find what data prep is still missing. For the lyric corpus specifically use eval-lyrics; for a trained checkpoint use eval-checkpoint.

2026-07-124

run-tests

diskrotrepo/nano

Run the nano pytest suite correctly. Use this skill when the user wants to run the tests, run a specific test file, run or skip the slow benchmark tests, or is writing or debugging a test and needs the fixtures and conventions.

2026-07-124

name	eval-training-run
description	Evaluate the health of a LIVE nano training run from its Modal logs — loss trajectory vs the random floor, per-codebook convergence order, val-checkup trend and strikes, throughput stability, ETA, and infra noise triage. Use this skill when the user asks "how is training going", "is the run healthy", "evaluate the training run", "is it diverging", or wants an ETA to a step count. For evaluating a finished CHECKPOINT (sample quality, WER, overfitting) use eval-checkpoint instead; for launching/resuming runs use train-model.
allowed-tools	Read, Bash

Evaluate a live training run

Gather

modal app list | grep nano-train                  # find the app id + state
APP=ap-...
modal app logs $APP 2>&1 | grep -aE "step +[0-9]+/" > /tmp/steps.txt
modal app logs $APP 2>&1 | grep -aE "checkup at step|early stopping" > /tmp/checkups.txt
modal app logs $APP 2>&1 | grep -acE "Timed out .* waiting for clients|Traceback|CUDA out of memory"

(Full-log queries can hit a server-side resource limit once CLAP progress spam accumulates — always pipe through a grep filter, never page raw logs.)

Judge — in this order

Reference numbers: random-chance loss is ln(vocab) = ln(1024) ≈ 6.93 per codebook — vocab is 1024 for both DAC and SpectroStream, so the floor survives the codec switch (re-derive it if vocab ever changes). Train loss above 6.93 after warmup = confidently wrong, not "still learning".
Val trajectory (the verdict): checkups every 1000 steps must trend DOWN. With EMA on (v9 default, use_ema=True) the checkup score — and best.pt — are the EMA weights, not the live ones: (a) EMA smooths noise away, so consecutive worse checkups are a stronger divergence signal than they were on v8; (b) don't compare the step-line train loss (live weights) against the checkup score (EMA) as if they were the same basis. Two+ consecutive strikes while LR is still climbing through warmup = the 2026-06-12 divergence signature — stop the run, don't wait out patience. Healthy v9_stereo reference (B200:4, launched 2026-07-12): every checkup a new best through 13k — 5.2321@11k, 5.1777@12k, 5.1300@13k (just past the 10k warmup end, full lr 1.5e-4). Healthy v8 reference (bf16+qk-norm): 6.75@1k, 6.69@2k, 5.11@8k, 4.93@10k (warmup end) — every checkup a new best. Diverged v8 reference: 7.17 → 7.87 → 8.70 by step 3000. Scores are NOT comparable across vintages (different codec + K). The step line's grad field is the leading indicator: healthy is FLAT — v9_stereo sits ~0.10–0.20 at full LR; the v8 recipe ran a 0.3–2.5 band (clip=1.0 occasionally touched is fine on either). A ramp across windows precedes divergence by thousands of steps.
Per-codebook order: shallow codebooks converge first, so a checkup's cb[...] spread must be ordered low→high from cb0 (healthy v9 @13k: cb0 1.88 rising monotonically to cb23 6.46). Adjacent levels swapping in a single noisy 25-step train line is normal; judge ordering on checkup lines (averaged over 50 eval batches). cb0/the shallow levels HIGHEST or climbing = the divergence canary — cb0 led the 2026-06 explosion by ~800 steps. All K equal after thousands of steps = delay-pattern suspicion.
Throughput: reported tok/s counts global-batch × K × frames, so it is NOT comparable across codecs or shapes. v9_stereo steady state ≈ 2.95–3.05M tok/s on B200:4 (the hardwired multi-GPU config; ~3.46M tokens/step → ~0.86 steps/s, ~29s per 25-step line). v8-era: ~1.6–1.7M on 8×H100 (K=9, 60s). Isolated low windows in the first few thousand steps = late first-hit variant compiles (finite, self-quiescing). Persistently low/erratic after that = a starving rank. Ignore the step-25 line (compile-polluted average).
ETA: steps/s from the spacing of recent step lines (each line = 25 steps), then (target - current) / steps_per_s. State it with the caveat that val-driven early stopping (patience 20) may end the run first.
Infra events: count NCCL rendezvous timeouts ("waiting for clients") — one is a transient (Modal auto-retries; resume is checkpoint-safe), repeats on the same app = bad node, stop + relaunch for fresh placement. Check the benign-noise list before flagging anything else.
Checkpoint state: modal volume ls nano-ckpts <subdir>/ — best.pt updates on checkup improvements (EMA weights when EMA is on), latest.pt/step_*.pt every 5000 steps.