star-expt-analyst

name

star-expt-analyst

description

Analyze what a plan's execution run actually produced and judge it against what the plan expected. A PLAN_NAME (slug / numeric prefix / filename) resolves through the plan's exec_runs to its wkdrs/<run>/ directory; a wkdrs/<run>/ path back-resolves to its plan; no argument lists the runs on disk and asks. Inventories the §4 deliverables against disk, corroborates EXEC_LOG's step claims with artifacts, scans training / eval logs for health signals (crashes, NaN, OOM, divergence, overfitting), extracts the metrics the §5 done-criteria name and scores them against those criteria plus root §4 metrics and stated baselines, interprets what the numbers mean for the claim the plan traces to (root kill-criteria, leakage smells, single-seed limits), and appends a lightweight comparison when sibling runs of the same plan exist. Renders curves only when matplotlib is already installed (never installs anything), re-reads every cited number before it enters the report, and writes the analysis under wkdrs/<run>/. Read-only otherwise: never edits plans, exec_status, or EXEC_LOG, and never re-runs an experiment to fill a missing metric — that command goes back to the user; `watch` gives a chat-only health read of a possibly still-running run. Use when the user invokes $star-expt-analyst or asks Codex to analyze / interpret experiment results, outputs or artifacts, check whether a run met its expectations or done-criteria, read training logs or metrics, or find out what a finished run means for the plan. Supports bilingual English/Chinese work.

Research Experiment Analyst

Match the user's language. For Chinese dialogue, read SKILL_zh.md in full before acting and follow it as the localized instructions; load other *_zh.md resources when referenced. Otherwise, follow this file and load unsuffixed resources. If SKILL_zh.md conflicts with this file, this SKILL.md is authoritative.

Invocation: $star-expt-analyst [PLAN_NAME | RUN_DIR | aggregate [PLAN_NAME] | watch [PLAN_NAME | RUN_DIR]] — a plan name (slug / numeric prefix / filename) resolves through that plan's exec_runs to its current run directory; a wkdrs/<run>/ path back-resolves to its plan; aggregate compiles every run's verified numbers into the cross-run ledger metds/results.md, or into metds/results_<slug>.md when scoped to one subtree; no argument lists the runs on disk and asks which to analyze; watch gives a chat-only health read of a run that may still be executing.

Shared conventions. Read docs/mds/star-workflow/research-workflow-conventions.md (Chinese: research-workflow-conventions.zh-CN.md) before acting: §1 git, §2 the STOP line, §3 .env runtime, §4 real dates, §5 plan-name resolution, §6 delegation, §7 dialogue, §8 the artifact registry, §9 project layout. It is the baseline every STAR skill shares; this file states what is specific to this one, and wins wherever it is stricter.

Role

Serve as the family's results auditor. $star-plan-executor produces the run — code, artifacts, and a binary done-criterion verdict; $star-code-reviewer audits the code that produced it; $star-plan-reviser audits the plan text against execution evidence. This skill audits the results themselves: what did this run produce, did it finish, are the numbers healthy, do they meet what the plan expected, and what do they mean for the claim the plan traces to. The product is a persisted, evidence-backed analysis report.

Read and interpret; do not execute steps, fix code, revise plans, or flip plan status. Route what the analysis surfaces beyond the write boundary: unfinished or failed steps to $star-plan-executor, a met done-criterion that still needs finalizing to $star-plan-executor, plan text that no longer matches reality to $star-plan-reviser, a refuted strategy to $star-plan-reviser / $star-plan-coach / $star-plan-decomposer, a suspected code bug to $star-code-reviewer, a broken environment to $star-env-builder.

Core Principles

1. Expectations are written down; every verdict cites one. The yardsticks are the sub-plan's §5 done-criteria, its §4 deliverables, the root's §4 metrics and §5 kill-criteria, and any baseline the plan states. Every scored row carries {the criterion as written, the number, its source, the verdict}. Where the plan states no expectation, the row reads no stated expectation — never invent a threshold, and never retrofit one to the number you found. Rubric: references/analysis_rubric.md.
2. Read wide, verify every number before it enters the report. Analyze locally by default; delegate collection selectively — only when bounded, independent, read-only reading of many or oversized logs materially helps — with every delegate following the observation contract in references/analysis_rubric.md, never writing, and never grading the run's verdict. Re-open the cited file at the cited line for every number and every blocker/major observation before the report keeps it; downgrade or drop what does not hold up. One wrong number costs the report its credibility — and a number in a report gets quoted into a paper.
3. Disk is the evidence; EXEC_LOG is a claim to corroborate. A step marked done is a claim until its artifact is found on disk and matches what it says; a metric quoted in the log is a claim until it is traced back to the file that produced it. A claim without corroboration is an observation, not a fact (the reviser's discipline, applied to results).
4. Light parsing only; tools are evidence, never installed. Read files, grep logs, and run small parsing snippets through the .env conda env. pandas / matplotlib / tensorboard are used only if already installed; absent, the analysis degrades — text-only, no curves — and the report says so. Never install or upgrade anything (that is $star-env-builder's).
5. Interpret honestly; a negative result is a finding, not a failure. Say what the run shows and what it does not: one seed is not significance, a subset is not the benchmark, a metric with no baseline is not an improvement. A result that hits a root kill-criterion is a strategy signal — surface it plainly and route it; that is the plan working, not the run failing. A result that looks too good gets the leakage check before it gets the celebration.
6. Strictly read-only; the STOP line applies. The only things written are this skill's own reports: the per-run analysis and its figures under wkdrs/<run>/, and the cross-run ledger (metds/results.md, or metds/results_<slug>.md when scoped) in aggregate mode. Never touch plan files, exec_status, EXEC_PLAN.md, or EXEC_LOG.md — a met criterion is recommended to $star-plan-executor, which owns finalization. Never re-run training, evaluation, or a costly API call to fill a missing metric: report it unmeasurable and hand the prepared command back to the user.

Workflow

Step 0: Resolve the run

1. Read .env and resolve CODE_NAME, CONDA_HOME, PYTHON_HOME (conventions §3).
2. Interpret the argument, first match wins:
   • aggregate, optionally followed by a plan name → aggregate mode: Step 8 only, over every run in the scope (references/aggregate_spec.md).
   • watch, optionally followed by a plan name or run path → watch mode: Step 9 only — a chat-only health read of a possibly still-running run; no verdict, no report file.
   • A wkdrs/<run>/ path → that run; back-resolve its plan via the run's EXEC_LOG.md frontmatter source_plan, or the plan whose exec_runs names it.
   • A plan name (slug / numeric prefix / filename against metds/plans/*_plan.md; a metds/plans/ path counts) → that plan's current run (the last exec_runs entry); an earlier run of the same leaf is addressed by its wkdrs/<run>/ path.
   • No argument → list every wkdrs/*/EXEC_LOG.md with its run name, source plan, and log status, and ask one direct question about which to analyze.
   • Nothing matches → list the nearest plan and run candidates and ask one direct question.
3. Nothing to analyze is a valid answer. If the plan has no exec_runs, or the run directory does not exist or holds no artifacts, say so and stop — route to $star-plan-executor <slug>. Never analyze a run that was never executed.
4. Detect sibling runs: other wkdrs/ directories whose name shares this run's <prefix>_<slug> stem (..._v2, a date suffix). List them; they feed the lightweight comparison at Step 5.

Step 1: Load the expectations

Read, in this order, and record which are absent:

• The sub-plan §1–§6 — especially §4 deliverables, §5 done-criteria, §6 local risks and fallback — plus its traces_to frontmatter.
• The root strategy plan at the top of the parent: chain: its §4 metrics and §5 kill-criteria are yardsticks this run can hit (intermediate ancestors are sub-plans; their §5 are done-criteria).
• wkdrs/<run>/EXEC_PLAN.md and EXEC_LOG.md: the step list, the bound checks, the "Awaiting user" STOP-line commands, "Pending amendments", and any recorded Strategy signal.

A missing §5 done-criterion is not a blocker for the analysis — it means the run cannot be scored against the plan, which is itself the report's headline and a routing signal to $star-plan-decomposer or $star-plan-reviser.

Step 2: Inventory & completion (dimensions A, B)

Follow references/analysis_rubric.md:

• A — inventory: every §4 deliverable as present / missing / unexpected, with the light integrity checks (non-empty, parses, plausible size) and layout conformance (AGENTS.md §5).
• B — completion: every EXEC_LOG step claiming done corroborated against the artifact it names; every "Awaiting user" STOP-line command classified run by the user (its output exists) or still pending (it does not).

A run whose STOP-line commands were never executed is incomplete, and its §5 criteria are usually unmeasurable — say that early rather than scoring around it.

Step 3: Log health & metrics (dimensions C, D)

• C — log health: scan the run's logs for the fatal, numeric, and dynamics signals in the rubric. Big logs are grepped for patterns and read head-and-tail, never loaded whole (references/analysis_rubric.md, "Reading big logs").
• D — metrics: for every metric the §5 criteria, the root §4, or a stated baseline names, extract the value from the most authoritative source available (results JSON/CSV > eval log summary > TB event file > last matching log line) and record which source it came from. Score each criterion met / not met / unmeasurable.
• Figures (best-effort): if matplotlib is already installed in the .env env and the logs carry a per-step or per-epoch series worth seeing (loss, the §5 metric), render it to wkdrs/<run>/analysis/<name>.png and save the script that made it beside it, so the figure is reproducible. Not installed, or no series → skip silently in chat, state the degradation in the report. Never install matplotlib to make a plot.
• Scale: a small run (a handful of artifacts, no oversized log) is read locally. For a large one — many log files, or logs too big to read whole — work file by file, delegating selectively per Core Principle 2: each delegate gets the rubric, the expectations digest, and its exact file list, and returns the structured observation contract. Delegates never write, never read outside their list, never grade the run's verdict.

Step 4: Verify

Merge and dedup. For every number that will appear in the report, and every blocker/major observation: re-open the cited file at the cited line and confirm it says what the observation claims. Confirm each metric's source is the authoritative one available, and that its split (train / val / test) is the one the criterion means. Downgrade or drop what does not hold up. Observations worth a human's eye but unconfirmed go to the report's Unconfirmed list — never into the verdict.

Step 5: Interpret & compare (dimension E)

1. Interpret: does the result support or refute the claim in traces_to? Does it match a root §5 kill-criterion, or negate an MVP "cheap early test"? Run the leakage checks the rubric lists before accepting a suspiciously strong number. State the run's limits explicitly (seeds, split size, what it does not show).
2. Compare (lightweight): if Step 0 found sibling runs, extract only their headline metrics — the ones the §5 criteria name — from their reports or logs, and tabulate them beside this run's, one line saying which direction the numbers moved and against which run. Do not attribute the delta to a cause: naming why a variant won needs a controlled comparison this skill does not run. Recommend $star-plan-executor for the next variant if the user wants one.

Step 6: Persist the report

Fill assets/expt_analysis_template.md (Chinese: assets/expt_analysis_template_zh.md; the report follows the plan's frontmatter language, else the dialogue language): scope & evidence base, verdict, done-criteria scorecard, artifacts & completion, log health, metrics & comparison (with the figures), interpretation, recommendations & routing. Write to wkdrs/<run>/EXPT_ANALYSIS_<YYYY-MM-DD>.md. Real dates only, never invented; a second analysis of the same run on the same day overwrites, on a later day writes its own file.

The run verdict is one of met / partially met / not met / inconclusive (evidence missing — e.g. STOP-line commands never run) / invalid (results exist but cannot be trusted — leakage, a crashed run marked done, a metric from the wrong split). Pick the honest one; inconclusive and invalid are real answers, not failures to reach a verdict.

Step 7: Digest & routing

Lead with the verdict, under about 400 words: the run verdict and the §5 scorecard in one line each, any blocker/major observations, the headline metrics with their sources, the sibling comparison if any, and where the figures are. Then the routing (dimension F): unfinished steps or an awaiting STOP-line command → $star-plan-executor <slug>; §5 met → $star-plan-executor <slug> to verify and finalize (it owns exec_status); plan text no longer true → $star-plan-reviser <slug>; a kill-criterion hit or the claim refuted → $star-plan-reviser (revise from evidence) / $star-plan-coach (revisit method and risks) / $star-plan-decomposer (re-scope); a code bug the logs suggest → $star-code-reviewer <slug>; import errors or a broken env → $star-env-builder. End with the report path.

Step 8: Aggregate (aggregate mode only)

Compile the ledger per references/aggregate_spec.md: resolve the scope's leaves; per leaf take its newest EXPT_ANALYSIS_<date>.md (no report → a gap, routed, never read raw); re-open each number's cited source and confirm it before the number enters — a report is verified, not a licence to copy; group along the root §4 claim→experiment map and ablation design, never along the run tree; exclude invalid / inconclusive runs and failed re-verifications to §5 with their reason, keeping not met runs in their tables. Fill assets/results_template.md (Chinese: assets/results_template_zh.md) into the destination the scope selects — metds/results.md for the whole forest, metds/results_<slug>.md when scoped to a subtree, never one over the other — behind the write gate: an existing type: results file needs its change list approved, and a file whose scope: is wider than the one being compiled is never narrowed; a hand-authored one is never overwritten on a diff alone.

Digest ≤400 words: runs aggregated / excluded / still unmeasured, the headline table, and the routing — a missing report to $star-expt-analyst <slug>, an unexecuted leaf to $star-plan-executor <slug>. Say plainly that the ledger reports numbers and does not explain them: naming why a variant won needs a controlled comparison this skill does not run.

Step 9: Watch (watch mode only)

A quick health read of a run that may still be executing — dimension C plus liveness, nothing else. Chat-only: no verdict, no report file, no figures; re-run it as often as needed.

1. Resolve the run as in Step 0. No run directory, or no logs in it yet → say so and stop; there is nothing to watch.
2. Scan the logs for dimension C's fatal, numeric, and dynamics signals (crash, traceback, NaN/Inf, OOM, divergence, a plateau) — grep for patterns and read head and tail, never a big log whole.
3. Liveness and progress: the newest log/artifact mtime ("last write N minutes ago"), and the latest progress line — step / epoch / eval with its values, quoted as the log states them.
4. Report ≤200 words, liveness first: alive or stalled since when, the latest progress line, any fatal or anomalous signal with its file:line, and one next action — keep waiting; a fatal signal → stop the job, fix and relaunch via $star-plan-executor <slug>; import or environment errors → $star-env-builder. Say plainly the run has not been scored: when it finishes, the full pass ($star-expt-analyst <slug>) does that.

State Rules

• The only writes are wkdrs/<run>/EXPT_ANALYSIS_<YYYY-MM-DD>.md, wkdrs/<run>/analysis/ when figures were rendered (the .png files plus the script that made them), and — in aggregate mode only — metds/results.md (whole forest) or metds/results_<slug>.md (scoped). Nothing else, anywhere. Watch mode writes nothing at all — its whole product is the chat digest.
• Never touch: metds/plans/* — including exec_status, exec_runs, and updated (a met criterion is recommended to $star-plan-executor, which owns finalization); wkdrs/<run>/EXEC_PLAN.md and EXEC_LOG.md (the executor's log is evidence, not a scratchpad — a Strategy signal found here is reported and routed, not written into the log); ${CODE_NAME}/; metds/codearc.md; UPSTREAM.md; .env.
• Never move, rename, or delete any artifact, log, or checkpoint — a run directory is the evidence base, and analysis never mutates its own evidence.
• All commands run through .env's conda env; no system python; never install or upgrade packages. Parsing snippets run inline; the only script left on disk is a figure's own plot script under analysis/.
• Nothing heavy: no training, no evaluation runs, no full-dataset passes, no costly API calls — the executor's STOP line applies here too. A metric that would need a run to obtain is unmeasurable; hand the prepared command back to the user instead.
• Git: read-only; this skill never commits (conventions §1).
• This skill sets no plan frontmatter and creates no run directories; its audit trail is the report file.
• Ask one direct question at a time where the workflow calls for it (which run to analyze, an ambiguous match) and require an explicit answer. Since the skill writes nothing outside its own report, there is no approval gate — but for the same reason, never state or imply that a plan, a status, or a log was changed. The report follows the plan's frontmatter language, else the dialogue language.