baseline

# Baseline This skill establishes the reference system the quest will compare against. The target is one trustworthy baseline line, not an endless reproduction diary. ## Interaction discipline - Follow the shared interaction contract injected by the system prompt. - Keep ordinary setup and debugging updates concise. - Use richer milestone updates only when the baseline becomes trusted, caveated, blocked, waived, or route-changing. - Codex-native execution boundary: use Codex-native tools for routine setup inspection, Git mechanics, dependency checks, and non-evidence verification. Use `ds_bash_exec` for baseline setup, reproduction, monitoring, or verification commands whose logs must be durable DeepScientist evidence. - Prefer `ds_bash_exec` for setup, reproduction, monitoring, and verification commands so the baseline line stays durable and auditable. ## Tool discipline - **Do not bypass DeepScientist semantic tools for durable research state; Codex-native file/shell/process tools remain appropriate for routine operation-layer work.** - **Use Codex-native execution for routine shell/CLI/Python/node/git/npm/uv/environment checks. Use `ds_bash_exec` for baseline setup, reproduction, monitoring, or verification that must be durable quest evidence.** - **For git work inside the current quest repository or worktree, prefer `quest-local git state via backend service or Codex-controlled git action(...)` before raw shell git commands.** - **If a generic git smoke test is needed outside the quest repo, use `ds_bash_exec ...)` in an isolated scratch repository.** ## Non-negotiable rules - no fabricated metrics, logs, run status, or success claims - do not skip baseline steps or silently simplify the route when that would change trust or comparability - do not claim a baseline is ready before verification is complete - do not infer missing commands, scripts, or parameters when the uncertainty could change the result - any unavoidable guess must be written down explicitly with expected impact - use web search for discovering papers or repos, but use `artifact.arxiv(paper_id=..., full_text=False)` for actually reading a source arXiv paper when it exists - set `full_text=True` only when the short form is insufficient - for Python baselines, environment setup should be standardized around `uv` ## Stage purpose The baseline stage should produce a usable reference point through one of four routes: 1. attach an existing reusable baseline 2. import a reusable baseline package 3. reproduce a baseline from source 4. repair a broken or stale baseline Keep the classic control flow: 1. analysis 2. setup 3. execution 4. verification These are control gates, not paperwork walls. ## Quick workflow 1. Read the source paper and source repo first, or record exactly what is missing and why. 2. Choose the lightest trustworthy route: attach, import, reproduce, or repair. 3. Start with the fast path whenever the current baseline object, command path, and acceptance target are already clear enough to validate cheaply. 4. Before substantial baseline setup, code edits, or a real baseline run, create `PLAN.md` and `CHECKLIST.md`; short-form files are enough for simple fast-path work. 5. Keep one dominant phase visible: analysis -> setup -> execution -> verification. 6. Prefer one clean implementation pass, one smoke test, and then one normal baseline run. 7. Retry only when smoke, verification, or runtime evidence shows a concrete failure or incompatibility. 8. Close the stage by confirming or waiving the gate, then hand off with a concise `1-2` sentence summary of trust status and next anchor. ## Fast-path first Default to the lightest baseline path that can still establish a trustworthy comparison. Default to a fast path when it can establish trust with less work. Fast path is the default when any of the following is true: - `requested_baseline_ref` or `confirmed_baseline_ref` already points to the active baseline object - the route is clearly `attach` or `import` - the repo entrypoint, dataset or split, and metric contract are already concrete enough to validate cheaply - reproduction requires no meaningful code changes and the main uncertainty is only whether the command still runs Fast path means: - do not restart broad baseline discovery by default - do not front-load a full codebase audit when the entrypoint is already concrete - use a minimal `PLAN.md`, a minimal `CHECKLIST.md`, one bounded smoke test when needed, and then one real validation or run - default to reuse-and-verify when runtime already attached a concrete baseline Escalate from fast path to fuller audit only when: - the paper and repo disagree materially - the real run or eval entrypoint is unclear - code changes are likely required - the contract spans multiple metrics, datasets, subtasks, or splits that still need interpretation - the same failure class reappears after one documented autonomous fix - the quest is trying to publish a reusable global baseline rather than only clear the current gate ## Use when - no credible baseline exists yet - the current baseline is unverified or stale - the user already has a baseline package that should be attached or imported - a reproduction failed earlier and now needs repair - the quest resumed and the baseline trust state is unclear ## Do not use when - the quest already has a verified active baseline and the next move is ideation or execution - the user explicitly waived the baseline gate and that waiver is durably recorded ## Stage gate Do not proceed to comparison-heavy downstream work unless one of the following is durably true: - a baseline has been attached and accepted - a baseline has been imported and accepted - a baseline reproduction has completed and been verified - an explicit waiver decision exists with a clear reason Operationally: - call `artifact.confirm_baseline(...)` once the accepted baseline root and trusted comparison contract are clear - call `artifact.waive_baseline(...)` when the quest must continue without a baseline - attach, import, or publish alone do not open the downstream gate ## Required plan and checklist Before substantial baseline setup, code edits, or a real baseline run, create a quest-visible `PLAN.md` and `CHECKLIST.md`. - Use `references/baseline-plan-template.md` as the canonical structure for `PLAN.md`. - Use `references/baseline-checklist-template.md` as the canonical structure for `CHECKLIST.md`. - `analysis_plan.md` and `REPRO_CHECKLIST.md` remain acceptable compatibility alias files when an older quest already depends on them. - For fast-path attach/import/prebound validation or a simple reproduce path with no expected code changes, short-form `PLAN.md` and `CHECKLIST.md` are enough. - The plan should put the user's explicit requirements and non-negotiable constraints first. - Then record the chosen route, source identity, command path, expected outputs, acceptance condition, safe efficiency levers, main risks, and fallback. - If the route, commands, source package, fallback path, or trust judgment changes materially, revise `PLAN.md` before continuing. - Once the route is concrete, stop reshaping code and commands speculatively. Default retry discipline: - do not rerun the same unchanged smoke command just to reconfirm the same fact - treat one autonomous retry for the same failure class as the normal upper bound - if the same failure class appears again, switch explicitly into `repair`, record `blocked`, or route through `decision` ## Required durable outputs The baseline stage should usually leave behind: - a baseline directory under `baselines/local/` or `baselines/imported/` - `PLAN.md` and `CHECKLIST.md` - a verification note or report - command, config, environment, and metrics pointers - a baseline artifact - a confirmed baseline gate via `artifact.confirm_baseline(...)`, or an explicit waiver via `artifact.waive_baseline(...)` - an optional registry publication if the baseline is reusable beyond this quest For simple attach/import flows or a straightforward reproduce flow, do not stall just to precreate every optional note file. Useful optional notes: - `setup.md` - `execution.md` - `verification.md` - `STRUCTURE.md` when the layout is non-obvious ## File-by-file contract - `PLAN.md` or compatibility alias `analysis_plan.md` is the required route contract before substantial setup, code edits, or a real run; it should state the route, source identity, command path, expected outputs, acceptance condition, main risks, and fallback. - `CHECKLIST.md` or compatibility alias `REPRO_CHECKLIST.md` is the required living state tracker; it should show whether the baseline object, smoke decision, real run decision, and final accept / block / waive outcome are explicit. - `setup.md` is optional unless environment or layout choices are non-trivial; if used, record the working directory, environment route, important config paths, source revision, and notable setup deviations. - `execution.md` is optional unless the run is long, multi-step, or rerun-heavy; if used, record the launched commands, durable log paths, checkpoints, exit state, and any reruns or repairs. - `verification.md` is optional as a filename but required in substance before acceptance or blocked closeout; either this file or an equivalent report should record trusted metrics, expected-versus-observed comparison, caveats, canonical output paths, and the next anchor. - `STRUCTURE.md` becomes required when the workspace layout, mounts, symlinks, or generated outputs are non-obvious or meant for reuse; it should map the important directories and say which paths are canonical. - `attachment.yaml` is required for attached or imported baselines under `baselines/imported/`; preserve source identity, selected variant when relevant, and attachment provenance there. - `<baseline_root>/json/metric_contract.json` is the canonical accepted comparison contract; once the baseline is accepted, do not leave the authoritative metric surface only in chat, memory, or prose. - `Result/metric.md` is scratch-only; it may help during execution, but it is never the final source of truth. Minimum stability rules: - before the first real run, leave one durable note with the chosen route, expected command path, target outputs, and main risks - after each smoke test or real run, record what actually happened and whether the route still looks viable - before acceptance, leave a clear verification note and baseline gate decision - every accepted baseline should leave one accepted baseline artifact - every blocked baseline line should leave one blocked report and one next-step decision - if one rolling note is enough for a simple baseline line, use it ## Durable path contract Use the real runtime paths consistently. Quest-local paths: - reproduced baseline root: `<quest_root>/baselines/local/<baseline_id>/` - attached or imported baseline root: `<quest_root>/baselines/imported/<baseline_id>/` - attachment record: `<quest_root>/baselines/imported/<baseline_id>/attachment.yaml` - canonical baseline metric contract JSON: `<baseline_root>/json/metric_contract.json` - baseline artifact record: `<quest_root>/artifacts/baselines/<artifact_id>.json` - baseline reports: `<quest_root>/artifacts/reports/<artifact_id>.json` - confirmed baseline reference: `quest.yaml -> confirmed_baseline_ref` Global reusable registry paths: - baseline registry index: `~/DeepScientist/config/baselines/index.jsonl` - canonical baseline entry: `~/DeepScientist/config/baselines/entries/<baseline_id>.yaml` ## Baseline id and variant rules - `baseline_id` should be short, stable, and filesystem-safe - use letters, digits, `.`, `_`, or `-` - do not use spaces, `/`, `\\`, or `..` - if one codebase contains multiple comparable baselines, prefer one `baseline_id` with structured variants instead of inventing many near-duplicate entries - when variants exist, keep `default_variant_id`, `baseline_variants`, and per-variant metric summaries stable enough that later `experiment` and `write` stages can cite them directly Do not invent parallel durable locations when these runtime contracts already exist. Do not leave the authoritative metric contract only in chat, memory, or prose once the baseline is accepted. If a baseline is reproduced only because an analysis campaign needs an extra comparator: - still place it under the normal baseline roots - treat it as a supplementary analysis baseline unless the quest explicitly promotes it into the canonical gate - do not call `artifact.confirm_baseline(...)` for that supplementary case unless the quest truly intends to replace the canonical baseline ## Multi-baseline policy One quest may legitimately need more than one baseline. - explicitly mark which baseline is the primary downstream comparator - distinguish primary comparison baselines from fallback or infrastructure baselines - if several baselines are credible, record why the chosen primary baseline is the fairest paper-facing comparator - do not leave later stages guessing which baseline is authoritative ## Route order Prefer this order: 1. attach 2. import 3. reproduce 4. repair Prefer reuse over redundant reproduction. ## Workflow ### Phase 1. Analysis Before running anything substantial, determine: - exact task - dataset and split contract - metric contract - source baseline identity - source code path - expected run command or evaluation path - expected paper or repo numbers when they exist - local resource constraints Default analysis discipline: - read the source paper and source repo first - if runtime already exposes a matching `requested_baseline_ref` or `confirmed_baseline_ref`, validate that concrete object before restarting broad discovery - identify the real run or evaluation entrypoint - identify the dataset or split and metric contract - identify likely environment blockers - define the cheapest credible smoke test Escalate to a fuller audit only when the command path is unclear, the repo is large or confusing, repair mode is active, or custom code changes look likely. When the fuller audit is necessary, capture only what later stages truly need: - major entry scripts, configs, and modules - end-to-end data flow - evaluation path and metric computation path - obvious environment assumptions - obvious bottlenecks or incompatibilities If the source paper is available, record: - the core algorithm in compact, implementation-faithful form - the main reported numbers - the main weaknesses or bottlenecks likely to matter for this quest You may inspect local feasibility with shell-based checks for OS, GPU, CPU, RAM, disk, Python version, and whether `uv` is available. The analysis phase should leave behind a concrete plan rather than only conversational intent. ## Phase 2. Setup Prepare the selected route: - attach: validate the selected baseline id and variant - import: place the imported baseline metadata under the quest and confirm the package is readable - reproduce: prepare the baseline work directory, commands, config pointers, and environment notes - repair: identify the precise broken point before rerunning blindly For Python baselines, standardize environment setup around `uv`. ### Python environment rule: use `uv` - if the repo already contains `uv.lock` or a solid `pyproject.toml`, use `uv sync` - otherwise create a local virtual environment with `uv venv` - install dependencies with `uv pip install ...` - run setup, smoke tests, and real commands through `uv run ...` Practical rules: - prefer a quest-local or baseline-local `.venv` - prefer `uv run python ...` or `uv run bash ...` over relying on shell activation state - if a specific interpreter is required, make it explicit with `uv venv --python 3.11` or `uv run --python 3.11 ...` - if CUDA, PyTorch, JAX, or custom wheels require a special index URL, keep that install under `uv pip` - only accept a non-`uv` route when there is a concrete blocker that cannot be resolved locally Common `uv` patterns: - `uv sync` - `uv venv --python 3.11` - `uv pip install -r requirements.txt` - `uv run python scripts/smoke_test.py` - `uv run python train.py --config ...` Setup should record: - baseline id and source identity - working directory - config files - command template - expected outputs - known deviations from paper or source - the chosen `uv` route and Python version Fallbacks: - if Hugging Face access is blocked, record and try an approved local mirror such as ModelScope when that does not change the comparison meaning - if a quest already depends on `analysis_plan.md` or `REPRO_CHECKLIST.md`, keep the compatibility alias explicit rather than splitting truth across two active plans ## Phase 3. Execution Run only the work required to establish the baseline credibly. Execution rules: - keep commands auditable - keep logs durable - avoid uncontrolled side experiments during baseline establishment - checkpoint only explainable, minimal code changes - prefer equivalence-preserving efficiency gains such as larger safe batch size, cache reuse, checkpoint resume, and parallel downloads or workers - do not use an efficiency lever if it changes accepted baseline meaning, effective evaluation contract, or trust judgment Long-running execution discipline: - run one bounded smoke test before a substantial baseline reproduction - once the smoke test passes, launch the real baseline reproduction with `ds_bash_exec mode='detach', ...)` - monitor by forward progress instead of by short-window completion anxiety - do not report final success until the command actually finished and the expected result files exist - if you need to recover ids or inspect session state, use `ds_bash_exec mode='history')` or `ds_bash_exec mode='list')` - `ds_bash_exec mode='read', id=...)` returns the full saved log when it is `2000 lines or fewer`; for longer logs, inspect omitted middle windows with `start` and `tail` - during monitoring, prefer `ds_bash_exec mode='read', id=..., tail_limit=..., order='desc')`, and after the first read prefer incremental checks with `after_seq=last_seen_seq` - use `silent_seconds`, `progress_age_seconds`, `signal_age_seconds`, and `watchdog_overdue` as the default staleness clues - if a run is clearly invalid, wedged, or superseded, stop it with `ds_bash_exec mode='kill', id=..., wait=true, timeout_seconds=...)`, document why, and relaunch cleanly - do not let more than the `30-minute visibility bound` pass without a real inspection and a `next expected update time` - when the baseline code is under your control, prefer a throttled `tqdm` progress reporter and periodic `__DS_PROGRESS__` markers when feasible Keep retries bounded: - one smoke test is the default - one autonomous fix-and-retry for the same failure class is the normal upper bound - if the same failure class returns, stop looping ## Phase 4. Verification Verification is mandatory before baseline acceptance. Verify: - the run actually finished - the reported metrics came from the intended dataset and split - the metric definitions match the quest contract - the result is comparable to the paper, source repo, or selected target - any deviations are explicitly stated Classify the outcome as one of: - `verified_match` - `verified_close` - `verified_diverged` - `broken` Verification must explicitly separate: - likely implementation mismatch - environment mismatch - data or split mismatch - expected stochastic variance - unexplained divergence Verification should answer: - whether the baseline is trustworthy enough for downstream comparison - whether the result is reusable beyond this quest - whether another repair or rerun is justified - whether the line should stop here and hand off A verification report should be self-contained enough that a later stage can answer: - what was used - how it was obtained: attach, import, reproduce, or repair - what commands and configs were used - what metrics are trusted - what caveats remain - whether the result is reusable beyond this quest ## Baseline comparability contract The baseline stage is not complete just because something ran. It is complete when later stages can compare against it fairly. Before declaring a baseline usable, make the comparability contract explicit: - task identity - dataset identity and version - split contract - preprocessing boundary - evaluation script or evaluation path - required metric keys - metric directions - seed policy when relevant - source commit or source package identity - known deviations from the source reference Unless the user explicitly specifies otherwise, treat the original paper's evaluation protocol as the canonical baseline contract. If any of these fields are still materially unknown, do not pretend the baseline is a clean downstream reference. For the fuller checklist and verdict meanings, read `references/comparability-contract.md`. ## Feasibility and trust classes Before acceptance, classify feasibility as one of: - `full_reproducible` - `degraded_but_acceptable` - `blocked` And classify downstream trust as one of: - `verified` - `partially_verified` - `operational_but_incomparable` - `failed` Do not silently upgrade a degraded or merely operational result into a normal trusted baseline. ## Minimum baseline artifact content The accepted baseline artifact should include at least: - `baseline_id` - `baseline_kind` - `path` - `task` - `dataset` - `primary_metric` - `metrics_summary` - `environment` - `source` - `summary` If variants exist, also include: - `default_variant_id` - `baseline_variants` Metric-contract rules: - if the accepted baseline contract includes multiple metrics, datasets, subtasks, or splits, record all of them in `<baseline_root>/json/metric_contract.json` - keep `primary_metric` as the headline metric only; do not let it erase the rest of the comparison surface - when confirming a baseline, submit the canonical `metrics_summary` as a flat top-level dictionary keyed by the paper-facing metric ids - every canonical baseline metric entry should include `description`, either `derivation` or `origin_path`, and `source_ref` - if the paper reports both aggregate and per-dataset or per-task results, preserve both whenever feasible through `metrics_summary` plus structured rows rather than one cherry-picked scalar - if the source package already has a richer leaderboard table, structured result file, or `json/metric_contract.json`, reuse that richer contract instead of hand-writing a thinner one that keeps only one averaged scalar - `Result/metric.md` is optional temporary scratch memory only; reconcile against it before calling `artifact.confirm_baseline(...)`, but do not treat it as a required durable file ## Publication and reuse Use the registry deliberately, not as an afterthought. If the result is reusable beyond the current quest: - publish it through `a Codex-controlled baseline publish/admin action(...)` - ensure the payload includes identity, provenance, trusted metrics, and any variant structure - set `publish_global: true` only when verification is complete and reuse is justified If the current quest should reuse an existing baseline: - attach it through `artifact.attach_baseline(...)` - preserve the selected `baseline_id` - preserve the selected `variant_id` when one is used - keep the attachment durable under `baselines/imported/` If runtime state already includes `requested_baseline_ref` or a matching `confirmed_baseline_ref`: