| name | view-progress |
| description | Monitor a castform training run — status, reward scalars, and logs via the castform runs commands. Use after launching to watch a run or debug a stalled/failed one. |
View progress
Use this after launch to monitor the GPU run and debug stalled, failed, or
surprising reward curves.
Fast path
Track a run with the castform runs commands (all take --json):
castform runs status <run-id>
castform runs scalars <run-id> --mode eval --json
Status flows pending → active (running) → complete. Eval scalars are the
signal — reward should trend up and stay up. Every run is also viewable at
app.castform.dev/train/<run-id> (printed at launch).
Going deeper
Did training beat the baseline?
This is the real question — and there's no compare verb, so do it manually.
Your baseline is the eval reward you saw at castform validate (and step-0 of
the run) — for a held-out baseline, castform validate --train eval_dataset.jsonl
rolls out the eval split (there is no --eval rollout flag). Compare the eval
curve, not train — train reward can climb while eval flatlines (overfitting):
castform runs scalars <run-id> --mode eval --json
castform runs scalars <other-run-id> --mode eval --json
--mode is dynamic per run (train, eval, …); scalars defaults to train
when present, else the first available mode — so pass --mode eval explicitly for
generalisation. The human table is a latest-value summary; use --json for the
trajectory shape, peaks, and post-peak decline. To compare two runs (e.g. two
hyperparameter settings), read each one's eval scalars and diff them yourself.
Train reward can climb while eval falls. If eval peaks early, record the best step
and verify which checkpoint the platform can serve; the final checkpoint may be the
overfit one. A freshly-trained checkpoint can briefly lag in serving
(model_unavailable / a 502) — retry the read rather than assuming it failed.
Compare with external evals
External evals run an external/base model on the same eval examples. Compare them
to the trained model's eval mode, never to train reward.
Use the web UI when possible. From an agent, use the platform endpoints:
import httpx
from benchmax import config
from benchmax.platform.credentials import platform_bearer
run_id = "..."
c = httpx.Client(
base_url=config.platform_url(),
headers={"Authorization": f"Bearer {platform_bearer()}"},
timeout=60,
)
evals = c.get(f"/v1/train/runs/{run_id}/external-eval").json().get("evals", [])
eval_id = evals[0]["id"]
trained_avg = c.get(
f"/v1/train/runs/{run_id}/rollouts/mode-average",
params={"mode": "eval"},
).json()
external_avg = c.get(
f"/v1/train/runs/{run_id}/rollouts/mode-average",
params={"mode": "external-eval", "externalEvalId": eval_id},
).json()
comparison = c.get(
f"/v1/train/runs/{run_id}/rollouts/comparison",
params={"externalEvalId": eval_id, "page": 1, "limit": 20},
).json()
Use mode-average for the headline (trained_avg["avg"] vs
external_avg["avg"]) and /rollouts/comparison for matched examples
(modelGroups trained history, compGroups external eval). For one external
transcript, reuse the stored-rollout chain with mode="external-eval" and
externalEvalId=eval_id.
External evals usually have one batch/step, not a full trajectory. If the trained
model peaked before the final step, confirm which checkpoint can be served before
declaring whether it beat the external model.
Full run list + logs
castform runs list
castform runs logs <run-id>
Terminal states beyond complete: failed, stalled, cancelled,
out_of_credits, billing_error.
runs logs is not a stored-rollout transcript browser; it usually contains env
diagnostic/error logs. If reward looks odd, inspect stored rollouts instead.
Investigate stored rollouts
To read actual answers and per-component rewards from a completed run, use the
built-in rollout commands (no raw HTTP needed):
castform runs rollouts <run-id> --mode eval
castform runs rollouts <run-id> --example <EXAMPLE ID>
castform runs rollout <run-id> <ROLLOUT ID>
castform runs rollout <run-id> <ROLLOUT ID> --view
runs rollout joins the gold/ground truth back from your local
eval_dataset.jsonl (then train_dataset.jsonl) by prompt text — pass --dataset
to point at a specific file. Add --json to any of them for the raw payload.
These wrap the platform read endpoints (/rollouts/summary, /rollouts/heatmap,
/rollouts/<id>/details). For external-eval comparison (below) and per-step
component averages, drop to raw GETs with the SDK bearer:
import httpx
from benchmax import config
from benchmax.platform.credentials import platform_bearer
run_id = "..."
c = httpx.Client(
base_url=config.platform_url(),
headers={"Authorization": f"Bearer {platform_bearer()}"},
timeout=60,
)
If component-averages only shows latest-step data, use scalar histories such as
reward_stats/<component>/reward/mean or average details.rewards[] yourself.
Debug answer quality
When eval moves, read real answers rather than only scalar names:
- Bucket examples by correctness: fully correct, soft partial, wrong/empty.
- For RAG, decompose low citation recall: gold cited, gold retrieved but not cited,
and gold never retrieved. If correct answers cite retrieved valid sources, do not
reward-shape citations just because the aggregate recall is low.
- Separate lookup from multi-hop questions; multi-hop failures often need data/model
capability or retrieval changes, not a citation reward tweak.
- Check judge leniency: partial credit for vague, hedged, or question-restating
answers inflates pass@1 and weakens the training signal.
Background terminal monitors or polling loops do not survive a coding-agent/session
restart. After resuming a long run, re-run status/scalars rather than trusting an
old monitor.
Controlling / debugging a run
castform stop <run-id>
failed early → castform runs logs for an env/import/reward error; fix
main.py, re-validate, re-launch.stalled → the worker stopped reporting; check runs logs and the run URL.- Flat/odd reward → inspect stored rollouts, then go back to verify-environment
and audit the reward on transcripts before changing data or launching again.
