Finetune a pretrained KERMT encoder on a labeled CSV. The skill validates the input checkpoint (must be a pretrain ckpt — grover_base / cmim / hybrid), validates the labeled CSV, prepares the data (clean + features + optional split), then launches main.py finetune inside the kermt container (detached for hours-scale runs). Hyperparameters come from agent/config/defaults_finetune.json with per-flag CLI override.
Installation
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Finetune a pretrained KERMT encoder on a labeled CSV. The skill validates the input checkpoint (must be a pretrain ckpt — grover_base / cmim / hybrid), validates the labeled CSV, prepares the data (clean + features + optional split), then launches main.py finetune inside the kermt container (detached for hours-scale runs). Hyperparameters come from agent/config/defaults_finetune.json with per-flag CLI override.
license
Apache-2.0 OR CC-BY-4.0
compatibility
Requires docker, nvidia-container-toolkit, and a CUDA-capable NVIDIA GPU. Designed for Claude Code, Codex, and Nemotron.
Finetune a pretrained KERMT encoder on a user-supplied labeled CSV. The skill
is the workflow orchestrator: validate ckpt, validate data, prepare data,
launch the runner detached, return a run directory + container name.
Hardware requirements
GPUs: 1 (single-GPU). Finetune does not DDP; if you have multiple
GPUs visible, pass --gpus 0 (or whichever id) to select one. Multi-GPU
finetune is not currently supported.
VRAM: ≥ 8 GB for the default batch_size 32 configuration. Lower VRAM
works at smaller batch sizes — pass --batch-size N to override.
Disk: a few GB per run (checkpoint + features + logs).
Driver / CUDA: any host supporting CUDA 12.6 (the kermt image base).
kermt-setup validates this up-front.
Inputs
Required:
--ckpt <path> — input pretrain checkpoint (grover_base / cmim / hybrid).
The validator refuses already-finetuned ckpts with a redirect to
kermt-infer.
--csv <path> — labeled CSV. First column is smiles; every other column
is a target.
Optional:
--dataset-type {regression | classification | multiclass} — default
regression (from defaults_finetune.json). Drives loss, metric defaults,
and head initialization. For classification tasks pass
--dataset-type classification.
--targets COL [COL ...] — explicit target column names. If omitted, the
validator auto-detects numeric non-smiles columns and the skill confirms
with the user before proceeding.
--val-csv <path> and --test-csv <path> — user-provided val + test
splits. Either pass both or pass neither (the skill auto-splits using the
configured --split-type).
random and scaffold_balanced: build the val/test split internally
from the train CSV. No --val-csv / --test-csv needed.
index_predetermined: requires pre-split CSVs passed via
--val-csv + --test-csv (and, separately, per-fold index files —
see kermt/util/utils.split_data). Use this when the dataset ships
its own canonical split (e.g. tests/data/Biogen_for_grover/scaffold/ balance/<endpoint>/{train,val,test}.csv).
--metric NAME — mae (regression default), auc (classification default),
or any name kermt.util.metrics.get_metric_func accepts.
--epochs N / --batch-size N / --init-lr F / --max-lr F /
--final-lr F / --warmup-epochs F / --weight-decay F / --dropout F /
--bond-drop-rate F / --dist-coff F / --early-stop-epoch N /
--seed N — training-hyperparameter overrides. Anything not given is
filled from agent/config/defaults_finetune.json.
--ffn-hidden-size N / --ffn-num-layers N — shared FFN trunk dims.
--ffn-num-task-specific-layers N / --ffn-task-specific-hidden-size H —
per-target FFN heads (default 0 = off; useful for heterogeneous multi-target
finetunes). Both must be set together when N > 0.
--ensemble-size N / --num-folds N — multi-model / k-fold CV. Default 1
each.
--gpus 0 — single GPU id (default 0). Passing more than one is rejected.
--from-prepare <dir> — skip the prepare step and reuse an existing
prepare_data.json in <dir>. Useful when iterating on hyperparameters.
Workflow
Let $KERMT_REPO be the path to your kermt repo checkout, and assume
kermt-setup has built kermt:latest. All paths below are on the host; the
helper bind-mounts them at known container paths.
Pre-flight: ensure container + system probe.
$KERMT_REPO/agent/scripts/kermt_container.sh check_system | python -c "
import json, sys; d = json.load(sys.stdin)
if not d['ok']:
print('System check failed:', d['gaps']); sys.exit(1)
print(f'OK: {len(d[\"gpus\"])} GPU(s); CUDA via container toolkit')
"
If --targets was not given by the user, surface auto_detected_targets
from the JSON and ask the user to confirm before continuing. Abort on
ok: false.
Prepare the data (skip if --from-prepare given).
Pre-flight: check for sibling val.csv / test.csv. Before invoking
prepare_data, inspect the parent directory of <user-csv>. If a
canonical-looking sibling val.csv (or val_*.csv — common variants
include val_T.csv, val_clean.csv) AND a matching test.csv /
test_*.csv exist next to the train CSV, the dataset ships its own
pre-defined split. In that case set --split-type index_predetermined
AND pass --val-csv / --test-csv — otherwise the configured
split_type (default scaffold_balanced) will re-split the train CSV
from scratch and silently discard the user's val/test files. When in
doubt — or when the sibling files use non-canonical suffixes (_T,
_v2, etc.) — surface the situation to the user and ask which they
want.
Quoting target names. If any of the --targets column names
contain shell metacharacters (>, &, |, (, ), $, etc.),
single-quote each one when passing on the CLI to keep the shell from
eating part of the name. Example: --targets 'Log_Caco2_Papp_A>B' 'logD'. The CSV header itself is read directly by the downstream
trainer and is unaffected, but the prepare_data.json manifest's
targets[] field captures whatever the shell delivers — unquoted
metacharacters get truncated there.
Mount note:kermt_container.sh --data <host-csv> mounts the
parent directory of <host-csv> at /data. --val-csv and
--test-csv must therefore reference files in that same parent
directory. If val/test live in a separate directory (e.g. a sibling
splits/ folder), mount the parent of all three using --data <dir>
on a directory rather than a file.
Outputs land at $RUN_DIR/data/prepare_data.json. For scaffold_balanced
and index_predetermined, prep emits a single clean_full_csv + .npz;
the runner passes them through to main.py finetune which calls
split_data internally with the user-supplied seed.
Estimate runtime + echo applied defaults.
Finetune wall time is typically minutes-to-hours on 1 GPU.
Surface a summary of every flag that was filled from the defaults
vs user-supplied, so the user knows what was assumed. The runner
records this in args_applied.
Sample message:
"Filling from defaults_finetune.json: epochs=30, batch_size=32, split_type=scaffold_balanced. Override any of these with --<flag>."
Targets confirmation gate (hard requirement). Before launching the
runner, regardless of how the targets list was determined (CLI --targets,
auto-detection in step 4, or a user natural-language request like
"finetune on Caco2 and HLM"), echo the final targets list to the user with
an explicit count:
"Will finetune on N target(s): COL1, COL2, ...". If the user's request
specified a subset that doesn't match this list (e.g., they asked for 2
tasks via natural language but the list still has 4), treat it as a
discrepancy and re-prompt with the diff — never silently proceed on the
wrong target set. Wait for explicit confirmation before launching unless
--yes was given.
Launch the runner detached. (Consistent with the pretrain skills.)
$RUN_DIR/run.json (manifest with cmd_replay + image digest)
Log file: $RUN_DIR/logs/finetune.log
TensorBoard: $RUN_DIR/logs/tb (open with tensorboard --logdir $RUN_DIR/logs/tb)
Final checkpoints land at $RUN_DIR/ckpt/fold_0/model_0/model.pt
(best-val) and last_checkpoint.pt (sibling, auto-resume target).
Held-out test predictions + metrics land at
$RUN_DIR/ckpt/fold_0/test_result.csv. Paths vary with --num-folds
/ --ensemble-size.
To follow progress: kermt-monitor <RUN_DIR> (one-shot) or
docker logs -f <container-name> (streaming).
To block until the run finishes (useful for short test runs):
docker wait <container-name> — prints the exit code on completion.
Hard rules
Never modify the user's input ckpt. The runner passes its path via
--checkpoint_path; task/train.py loads it read-only into the model and
attaches a new FFN head. The source file stays untouched.
Arch comes from the ckpt, not from CLI/defaults. The runner extracts
hidden_size, depth, num_attn_head, activation, embedding_output_type,
self_attention (+ attn_hidden / attn_out when applicable) from the
ckpt's saved_args. There is no --hidden-size flag on this runner.
Never block on the long-running finetune. The skill launches via
run_detached and returns immediately after step 9. Use kermt-monitor.
Echo applied defaults back to the user. The args_applied field of
run.json records every flag's value + source (user / default-config).
Surface a one-line summary of every filled-from-default flag so the user
knows what was assumed.
Common errors
finetune_init requires a pretrain ckpt (grover_base / cmim / hybrid) →
the ckpt you passed is already finetuned (has task FFN heads). Pick a
pretrain ckpt instead, or use kermt-infer if you want to run
predictions with the existing finetuned model. To resume a finetune on
the SAME dataset, bypass the skill and call
python main.py finetune --checkpoint_path <ckpt> ... directly — the
agent skill doesn't support resume because saved-task identity
can't be machine-verified against the new training data.
prepare_data manifest reports ok=False → check errors for the failed
step (typically clean_smiles or save_features). Fix and re-run.
ffn_num_task_specific_layers=N>0 but ffn_task_specific_hidden_size is unset
→ MTL heads need an explicit hidden size. Pass --ffn-task-specific-hidden-size H.
finetune is single-GPU → multi-GPU finetune is not currently supported;
pick a single id.
Replayability
The run.jsoncmd_replay field is a single-line command that re-runs the
finetune with the same inputs, hyperparameters, and arch. To replay inside
the kermt container:
$(jq -r .cmd_replay $RUN_DIR/run.json)
If ok_to_replay: false in the manifest (because the kermt repo working
tree was dirty at launch time), the replay may not be bit-exact — pin the
exact commit via the repo.commit field and git checkout it
first.
