timesfm-train

name	timesfm-train
description	Autonomously fine-tune Google TimesFM 2.5 on crypto data with two-level Optuna search, Trackio logging, and end-of-round reflection. Use when the user says "autonomously train timesfm", "optimize timesfm on crypto until it stops improving", "run the autonomous trainer", or invokes `/train-timesfm`.

timesfm-train

Purpose

Runs a closed-loop LoRA fine-tuning search for TimesFM 2.5 over crypto data. An outer Optuna study sweeps recipes (target representation × feature set × time horizon); each recipe gets a fresh inner Optuna study over hyperparameters. After each round the reflection agent inspects the trial DB, Trackio alerts, and bad-trial logs, then proposes a bounded diff to workflow_state.json. The loop stops when a patience-based convergence check fires, or when the ralph-loop iteration budget is exhausted.

Prerequisites

• Consumer project directory (default: /local/home/cajias/Projects/nautilus/timesfm-crypto/) containing:
  • data/processed/ — raw OHLCV data ready for recipe-specific preprocessing.
  • scripts/finetune.py — the Trainer entry point that this package shells out to.
  • A uv-managed Python environment with timesfm, optuna, peft, and (optionally) trackio installed.
• Write access to runs/ under this package (or wherever --run-dir points).
• A clean git working tree in the trainer repo — every workflow-state change is committed.

Invocation

Interactive (recommended)

Start a ralph-loop over the slash command. Ralph re-feeds the command body each iteration until the completion-promise token is emitted:

/ralph-loop "/train-timesfm --consumer-dir /local/home/cajias/Projects/nautilus/timesfm-crypto" --max-iterations 50 --completion-promise NO_IMPROVEMENT_CONVERGED

Headless

The Python CLI runs the same orchestrator loop without the LLM-driven reflection step (uses a null-reflection that never diffs state):

uv run timesfm-train --consumer-dir <path> --max-rounds 20 --run-id <optional>

Headless mode is useful for sanity smoke tests and CI; real autonomous runs should use the ralph-loop path so the reflection agent participates.

Round Lifecycle

Each round is one iteration of the outer Optuna study. The orchestrator performs these five steps in order:

1. Load state. Read runs/{run_id}/workflow_state.json. Resume any in-flight outer study from outer_study.db.
2. Propose recipe. Invoke the recipe-proposer agent, which calls timesfm_trainer.recipe_study.suggest_recipe(state) and returns a fully-specified recipe. Preprocess to data/processed/{recipe_hash}/ if new.
3. Run hparam study. Invoke the trial-runner agent, which drives hparam_study.optimize(recipe, state). Inner trials report to Trackio; the MedianPruner kills stragglers. Best inner val-loss is reported back to the outer study via outer_study.tell(recipe, score).
4. Reflect. Invoke the reflection agent with the round's trial DB, Trackio alerts, and bad-trial logs. The agent returns a JSON object conforming to timesfm_trainer.reflection_protocol. Validated diffs are applied to workflow_state.json and git-committed; invalid diffs are discarded with a history note.
5. Judge. Invoke the convergence-judge agent which calls timesfm_trainer.convergence.should_stop(outer_study, state). If true, write the CONVERGED sentinel and emit NO_IMPROVEMENT_CONVERGED.

Whitelisted State Keys

Reflection may only modify these keys in workflow_state.json. Any other key in a proposed diff causes the diff to be rejected wholesale.

• patience — rounds without improvement before convergence (int, 2–20).
• epsilon — minimum val-loss delta to count as improvement (float, 1e-6 – 1e-2).
• inner_n_trials — hparam trials per recipe (int).
• inner_pruner_warmup_steps — MedianPruner warmup before it may prune (int).
• hparam_bounds.lr — log-uniform [min, max] for learning rate.
• hparam_bounds.lora_r — allowed values for LoRA rank.
• hparam_bounds.lora_alpha — allowed values for LoRA alpha.
• hparam_bounds.lora_dropout — uniform [min, max] for LoRA dropout.
• hparam_bounds.batch_size — allowed batch sizes.
• hparam_bounds.context_len — allowed context lengths (multiples of 32).
• hparam_bounds.weight_decay — log-uniform [min, max].
• hparam_bounds.warmup_ratio — uniform [min, max].
• hparam_bounds.epochs — allowed epoch counts.
• recipe_bounds.target_rep — allowed target representations.
• recipe_bounds.feature_set — allowed feature sets.
• recipe_bounds.time_horizon — allowed time horizons.
• dead_recipes — append-only list of recipe_hash values to skip.

Completion Tokens

• NO_IMPROVEMENT_CONVERGED — patience-based convergence fired. The winning adapter is at runs/{run_id}/recipes/{best_recipe_hash}/trials/{best_trial_id}/adapter/. Also written to runs/{run_id}/CONVERGED.
• BUDGET_EXHAUSTED_NOT_CONVERGED — ralph-loop hit --max-iterations before convergence. Written to runs/{run_id}/BUDGET_EXHAUSTED. The best-so-far artifact is still recorded in summary.md.

Artifacts

All artifacts live under runs/{run_id}/:

runs/{run_id}/
├── workflow_state.json
├── workflow_history.jsonl
├── outer_study.db
├── recipes/{recipe_hash}/{recipe.json,hparam_study.db,trials/...}
├── reflections/reflection_{round:03d}.md
├── reflections/reflection_{round:03d}_input.json
└── summary.md

Resuming

Point at an existing run to resume where you left off:

/train-timesfm --consumer-dir <path> --run-id 20260512-abc123

The orchestrator reads workflow_state.json, replays the outer study from outer_study.db, and starts at the next untried round. No state is lost; reflection sees the full history.

Safety Invariants

• Reflection cannot modify code, the Trainer implementation, or anything outside the whitelist above.
• All workflow_state.json mutations go through timesfm_trainer.workflow_state.apply_diff(), which validates the whitelist, appends to workflow_history.jsonl, and git-commits in the same call.
• If reflection's output fails protocol validation, the round proceeds unchanged and the failure is logged — the loop never hard-fails on bad LLM output.
• The completion promise NO_IMPROVEMENT_CONVERGED is emitted only when convergence.should_stop() returns true. The command body will not fabricate the token to escape the loop.

Design Reference

Full spec: /local/home/cajias/Projects/nautilus/timesfm-crypto/docs/superpowers/specs/2026-05-12-timesfm-autonomous-trainer-design.md