Menu
SkillsMP has collected 50 skills from open-thoughts/OpenThoughts-Agent. Open a skill to review its source and details.
Showing 40 of 50 collected skills.
Preserve + publish a finished RL (SkyRL/GRPO) training checkpoint after the job terminates (completed at max_steps OR early-stopped/scancelled) on an HPC cluster (Jupiter/Leonardo/Perlmutter). Covers: cancel pending retries, pick the BEST checkpoint by…
Post-run cleanup for a datagen (trace-generation) job on Iris/CoreWeave or an HPC cluster (Jupiter/Leonardo/Perlmutter): get the generated traces onto HF (penfever org) and free temporary disk. There is NO model checkpoint — the artifact is the trace dataset.…
Format HPC job-status reports as box-drawing tables, bucketed by job type (RL · SFT · Datagen · Eval · Catch-all), with the right metric columns, signal thresholds, and red-flags per bucket. Use whenever reporting active/recently-terminated job status —…
Launch, monitor, and manually clean up a trajectory-generation (datagen) job on Marin's Iris TPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, rescue, or kill a datagen/tracegen run on Iris.
Build a clean per-dataset summary table/CSV for a datagen (trajectory-generation) campaign — one row per task source with Status (COMPLETED / FAILED / RUNNING / NOT STARTED), N Trials Completed, Mean Turns/Trace, Mean Tok/Trace, Mean Reward, and the HF…
Analyze the token length of an OT-Agent conversation-format (ShareGPT-style) dataset — the per-trace distribution (median/p90/max) and/or counts under a token threshold + a metadata predicate (e.g. "task_complete AND < 32768 tokens"). Use when asked how long…
Given a list of models (HF name stubs) that have valid agentic ID eval scores in Supabase, build a ranking table: raw per-benchmark accuracy on the 3 ID benchmarks (SWE-Bench-100, OT-TBLite=dev_set_v2, Terminal-Bench-2.0=tb2), HF links to each eval's trace…
Run the Iris harbor job-history analyzer (scripts/iris/analyze_iris_harbor_job.py) on a datagen/eval job and read its JSON sidecar for trustworthy throughput / preemption / productive-trial stats. Use whenever a status check needs REAL metrics (gen tok/s,…
Run the full RL behavioral-analysis pipeline (scripts/analysis/analyze_rl_behavior.py) on a trained RL model to understand WHAT changed vs its pre-RL baseline, WHY, whether it PERSISTS, and its EVAL impact. Use when asked to "analyze RL behavior", "compare…
Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g. the delphi midtraining runs) — step progress vs target, loss/throughput, preemption + MAJOR step-gap detection, and checkpoint cadence. Use for an executor coordinator…
Build + push the datagen/eval TPU image (`ghcr.io/open-thoughts/openthoughts-agent:tpu`) — the Iris TPU runtime (vLLM-TPU 0.20.0 PyTorch/XLA+JAX + Harbor + the Daytona sandbox backend) — AS AN IRIS KANIKO JOB on CoreWeave `cw-us-east-02a` (a CPU-only amd64…
DESIGN a non-trivial codebase change (Harbor / MarinSkyRL / vLLM / OT-Agent / LLaMA-Factory) as a dependency-ordered STAGED PLAN before writing code — a feature port, a multi-step fix with parity requirements, a refactor, a kernel/perf change. Produces a…
EXECUTE a staged codebase plan (from code-create-staged-plan or an existing notes/<codebase>/ plan) one stage at a time, gate-by-gate, while keeping the local clone ground truth and a dated agent_logs/ progress log. For each stage: re-read the scope +…
Lint, run the pre-PR checks, commit, push, and author or update the branch's pull request in the required plain-text format. Use when committing, pushing, or creating/updating a PR.
Durably ARCHIVE everything informative from a finished run / experiment before it's cleaned up or its cluster artifacts age out — ALL Harbor trace_jobs (raw per-trial traces), ALL ray logs, ALL stdout/stderr (incl. vLLM/serving logs), and wandb. Pack-rat by…
Read, aggregate, and (carefully) write OT-Agent eval/model data in the Supabase registry. Use when asked to look up a model's ID/OOD benchmark scores, build/refresh an ablation or paper table from eval results, find unevaluated models, register a model/eval,…
Guardrailed DELETE of auto-registered eval `sandbox_jobs` rows that DID score but FAILED the harvest gate — partial evals (valid-complete <90% or non-benign infra-error >10%). These are the "DE-REGISTER candidate" rows the flawed_summ harvest defers. DISTINCT…
Safely purge stale, never-populated `sandbox_jobs` placeholder rows (eval launches that died/stalled before scoring) from the OT-Agent Supabase registry. Removes ONLY dead `Pending`/`Started` rows WE OWN that are >36h old with null…
Create a FRESH, snapshot-safe Harbor task dataset from an arbitrary input — a raw non-Harbor HF dataset (e.g. allenai/TMax-15K), a generator codebase (e.g. a GitHub repo like FrontierSmith), or just natural-language instructions with no seed data. The…
Launch a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter) via the OpenThoughts-Agent `hpc.launch --job_type datagen` entrypoint — the cluster-AGNOSTIC general flow: extract tasks from a parquet, then submit a managed vLLM-serve +…
Reduce the Daytona snapshot (unique-environment) count of a Harbor task dataset below the cap by editing its patcher's environment-build logic, without breaking task quality. Use when a dataset is flagged "SnapshotCapExceeded" / "N unique environments" with N…
Launch NON-AGENTIC (standard) data generation — plain vLLM/API completion generation with NO Harbor agent loop or Daytona sandboxes. Two paths: (1) Curator sharded datagen — the multi-node data-parallel run_curator_datagen_sharded.sbatch (one vLLM server per…
Debug a code bug with a structured debug log that records hypotheses, changes, and results.
Condense and clarify an operational/research doc (SKILL.md, ops.md, tracker, README, agent_log) OR the COMMENTS of a launch/config YAML (.yaml/.yml) by an editor-subagent dispatched with fresh context and a list of file paths to review. The editor MOVES…
Audit + recover a finished agentic eval. ALWAYS start with the read-only, idempotent completeness/health audit (§0): job finished? score present + non-zero + not obviously broken? HF traces present + linked? trial count ≈ n_rep × benchmark_size? — it writes…
Launch, monitor, and manually clean up an eval job on Marin's Iris TPU or CoreWeave H100x8 GPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, or kill a model evaluation (evalchemy / agent-harness benchmarks) on Iris.
Launch agentic Harbor evals through the OT-Agent unified eval listener (eval/unified_eval_listener.py) on any cluster: select models (query_unevaled_models.py / priority lists), wire the pinggy served-model tunnel, submit with the right preset + flags in…
Consolidate FINISHED standard / lm_eval (evalchemy) math-suite eval jobs — the Delphi #6279 MATH-500 / AIME24 / gsm8k grid launched via eval-standard-launch — into the SCORES.md tracker. Per job: confirm a NON-EMPTY seed42 result (a crash leaves it empty —…
Launch the fixed Delphi #6279 RL-scaling-laws downstream MATH eval suite (MATH-500 / AIME24 / gsm8k via evalchemy + lm_eval) on CINECA Leonardo, for completed SFT / RL / base checkpoints. Covers finding which cells are newly-completed-but-uneval'd, the…
File a GitHub issue for a bug or improvement found this session.
The PROCEDURE for one every-3-hours Iris job-status sweep — primarily the marin TPU datagen/eval jobs ("iris" here = the marin TPU cluster), plus CoreWeave GPU-RL as monitor-only. Query both clusters, run the harbor analyzer on each active TPU datagen/eval…
Produce a comprehensive cross-cluster job-status update for a recurring N-hourly cluster sweep. Gather squeue/sacct on each cluster (validating against false-drain), bucket every active + recently-terminated job by type (RL / SFT / datagen / eval /…
Re-register the every-3-hours Iris job-monitor cron (status check + datagen auto-rescue/keep-2-in-flight) if it has been lost. Primarily the marin TPU datagen/eval jobs ("iris" = the marin TPU cluster); also queries CoreWeave (cw-us-east-02a) GPU-RL as…
Re-create the local 3-hour tri-cluster cluster-sweep loop (Leonardo + CoreWeave(iris) + TACC(Vista); Jupiter SKIPPED until ~Jul 12) — the autonomous ML-ops monitor — if it has been lost. The loop is session-only and is dropped on any session restart, so…
Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a…
Launch / relaunch agentic MarinSkyRL (SkyRL GRPO) RL on Marin's Iris / CoreWeave GPU cluster (cw-us-east-02a, 8x H100-80GB + InfiniBand per node) via `python -m cloud.iris.launch_rl_iris` (run from the MarinSkyRL repo root) + the gpu-rl Docker image (NO…
Launch / relaunch agentic RL (SkyRL terminal_bench + Harbor + Daytona) on JSC Jupiter (GH200). Covers the dense 8B/32B FSDP2 arms (seqnorm, TIS, shaped, symclip, lrboost, loopshape) and the MoE/80B Megatron arms (Qwen3-Coder-30B-A3B, Qwen3-Next-80B-A3B) — the…
Deep single-RL-job health probe → a KILL / NO-KILL / ERROR recommendation for the supervisor. Dispatched as a subagent on every monitor tick for RL jobs in NEW/UNTESTED settings (new config/geometry/model, "debug" or "smoke-test" flavor, first launches after…
Preserve + publish a finished STANDARD (non-agentic GRPO) SkyRL RL checkpoint — the Delphi/rlvr/dapo math-and-reasoning cells launched via rl-standard-launch-leonardo (raw sbatch of hpc/skyrl_standard/leonardo/*, logger=console, NO Harbor/Daytona/trace_jobs).…
Launch, relaunch, or sweep STANDARD (non-agentic) SkyRL RL on CINECA Leonardo — GRPO on math/reasoning datasets (gsm8k, MATH/aime) and on-policy distillation (OPD, teacher→student) — via raw `sbatch` of the `hpc/skyrl_standard/leonardo/*` run scripts inside…