메뉴
Rank models per pred_len/seed for a dataset, producing MSE and MAE leaderboard CSVs. Use when the user wants leaderboards, rankings, or to compare model performance across prediction horizons.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | rank |
| description | Rank models per pred_len/seed for a dataset, producing MSE and MAE leaderboard CSVs. Use when the user wants leaderboards, rankings, or to compare model performance across prediction horizons. |
After experiments have produced performance.csv files under work_dirs/. Ask the user for the dataset name if not provided.
uv run python tool/rank_models.py --dataset <DATASET>
Writes model_rankings_mse.csv, model_rankings_mae.csv, and model_rankings_long.csv to work_dirs/<DATASET>/. Rankings are computed per (pred_len, seed) group; rank 1 = lowest metric value.
For a fair leaderboard (TFB-style), add --null-threshold 0.3 (exclude models missing on too many cells), --aggregate mean|median|max (collapse duplicates), and/or --fill-nan-with-mean.
Full flags, input requirements, and output formats: docs/en/rank-models.md (or tool/rank_models.py --help).
Add a new model to ModernTSF by scaffolding it with `tool/tsf.py new-model`, filling the architecture, and verifying with `tsf smoke`. Use when the user wants to integrate a new PyTorch model, register a custom architecture, or wire a new forecaster into the benchmark pipeline.
Verify model(s) end-to-end by running their smoke configs concurrently and reporting PASS/FAIL via `tool/tsf.py smoke`. Use when the user wants to check that a model (new or existing) trains and produces the right output shape, validate a port, or run a quick CI-style gate over many models.
Understand a model in ModernTSF by reading its README card as progressive disclosure — paper venue/date/arXiv/abstract first, then source only if needed. Use when the user asks "what is model X", "tell me about X", "了解 X 模型", "how does X work", which paper is X from, or wants to compare/learn what a forecaster does before running it.
Add or run a *probabilistic* forecasting model in ModernTSF — one that outputs quantiles or a distribution (prediction intervals / uncertainty) instead of a single point, and is scored with CRPS / WQL / coverage. Use this whenever the task involves prediction intervals, quantiles, uncertainty, a probabilistic or distributional forecast, pinball / quantile loss, Gaussian NLL, or the metrics `crps` / `wql` / `coverage_80` / `width_80` — even if the user only says "predict a range" or "give me uncertainty bands". Do NOT use the plain `add-model` flow for these: it builds a point model. Probabilistic output is a separate `output_type` axis, orthogonal to `task.mode`.
Package a finished run into a TSEval Submission Report (result + agent trajectory + report) and contribute it via a GitHub PR to the TSEval leaderboard repo, via `tsf trace` / `tsf submit`. Use when the user wants to submit results to the TSEval leaderboard, capture an experiment trajectory, or build a leaderboard from submissions.
Diagnose and report a ModernTSF framework defect upstream, with user approval, as a GitHub issue or a small verified PR against Diaugeia/ModernTSF. Use after reproducing a crash in src/ or tool/, wrong output or shapes, a broken config or registry, a doc/CLI mismatch, or another repository defect rather than a problem in the user's data or code.