菜单
Use when a quest is ready for a concrete implementation pass or a main experiment run tied to a selected idea and an accepted baseline.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | experiment |
| description | Use when a quest is ready for a concrete implementation pass or a main experiment run tied to a selected idea and an accepted baseline. |
| skill_role | stage |
Use this skill for the main evidence-producing runs of the quest.
cs_artifact_record payload={'kind': 'milestone'}, reply_mode='threaded', ...) report rather than another short progress line.sage-clay: #E7E1D6, #B7A99A, #7F8F84 for the default external messaging summary lookmist-stone: #F3EEE8, #D8D1C7, #8A9199 for conservative summariesdust-rose: #F2E9E6, #D8C3BC, #B88C8C for secondary comparisons onlypdf or svg) alongside the summary png.figure-polish/SKILL.md and follow its render-inspect-revise workflow before treating the image as final.cs_bash_exec for formal experiment commands, real runs, baseline/reproduction/ablation evidence, or paper-facing validations whose logs must become CodexScientist research evidence.cs_bash_exec for experiment commands so each run gets a durable session id, project-local log folder, and later read/list/kill control.cs_bash_exec for formal experiment runs, baseline/reproduction/ablation evidence, and paper-facing validations that require project-local provenance.project-local git state via backend service or Codex-controlled git action(...) before raw shell git commands.cs_bash_exec ...), not native shell tools.The experiment stage should turn a selected idea into auditable evidence. It should preserve the strongest old experiment-planning and execution discipline:
The experiment stage is not just "run code". It is the stage that converts an idea contract into evidence that other stages can trust. It is also the stage that should decide the next route once the measured result exists. Within the user's explicit constraints, maximize valid evidence per unit time and compute. Prefer equivalence-preserving efficiency upgrades first: larger safe batch size, mixed precision, gradient accumulation, dataloader workers, cache reuse, checkpoint resume, precomputed features, and smaller pilots. If a proposed efficiency change alters optimization dynamics, effective budget, or baseline comparability, treat it as a real experiment change and record it as such.
Use references/evidence-ladder.md when deciding whether the current package is merely executable, solid enough to carry the main claim, or already in the stage where broader polish is justified.
Completing one main run is not quest completion. After reporting the run, keep moving to iterate, analyze, write, or finalize unless a genuine blocking decision remains.
When the quest is algorithm-first, treat experiment as the execution surface of optimize, not as the terminal goal of the workflow.
After a measured result, the default next move is frontier review and optimize-side route selection rather than paper packaging.
Treat this as the short run-order summary. The detailed run contract, execution rules, and recording rules remain in Workflow.
1-2 sentences and confirm the baseline comparison contract.PLAN.md and CHECKLIST.md.run/* branch/worktree for this main experiment line; one durable main experiment should map to one run branch and one Canvas node.PLAN.md to lock the concrete run path, and use CHECKLIST.md as the living control surface while planning, implementing, pilot testing, running, and validating.cs_bash_exec.1-2 sentence summary that says what was tested, whether performance improved / worsened / stayed mixed, and the exact next action..py file first and execute it; reserve inline heredoc commands for short, simple snippets only.run/* branch.cs_artifact_record_main_experiment(...), route from the measured result:
optimize or decision for frontier review before launching another large run.Before a main run starts, confirm:
run/* target branch or isolated worktree for this exact main experimentIf any of these are materially unknown, stop and resolve them through decision.
Before substantial implementation work or a real main run, create a quest-visible PLAN.md and CHECKLIST.md.
references/main-experiment-plan-template.md as the canonical structure for PLAN.md.references/main-experiment-checklist-template.md as the canonical structure for CHECKLIST.md.PLAN.md should lead with the selected idea summarized in 1-2 sentences, put the user's explicit requirements and non-negotiable constraints first, and then make the run contract concrete: baseline and comparability rules, safe efficiency levers, code touchpoints, minimal code-change map, smoke / pilot path, full-run path, fallback options, monitoring and sleep rules, expected outputs, and a revision log.CHECKLIST.md is the living execution list; update it during planning, implementation, smoke testing, main execution, validation, and every material route change.PLAN.md before spending more code or compute.RUN.md, summary.md, and artifact payloads remain required outputs, but PLAN.md and CHECKLIST.md are the canonical planning-and-control surface before and during execution.PLAN.md makes the implementation route concrete, do not keep reshaping code and commands speculatively. The normal default is one bounded smoke or pilot run and then one real run, with retries only after a documented failure, invalidity, or new evidence that changes the expected outcome.Only modify the active project workspace for this experiment line.
Use:
cs_bash_exec session ids, progress markers, and exported logs from the actual runDo not claim run success without durable outputs.
A meaningful experiment pass should leave behind:
artifacts/experiment/<run_id>/ or the quest-equivalent canonical locationartifact_manifest.jsonrun_manifest.jsonmetrics.jsonmetrics.mdsummary.mdrunlog.summary.mdbash.logRecommended additional files:
claim_validation.mdrun_manifest.json should capture at least:
run_idIf a command needed for environment capture is unavailable, record that gap in the manifest and summary.
Before implementation or execution, state:
run_idauxiliary/dev or main/testPrefer to write this contract first in PLAN.md using references/main-experiment-plan-template.md, then keep the current execution state visible in CHECKLIST.md using references/main-experiment-checklist-template.md.
For substantial runs, also record the following seven experiment fields early and keep them updated during execution:
If the run contract changes materially later, record the change durably.
Treat the run contract as a research question contract, not only an execution checklist. Before coding, be able to explain:
If multiple candidate experiment packages exist, prefer the one with the best balance of:
Do not choose a package only because it sounds ambitious.
For paper-facing lines, default to this evidence ladder:
auxiliary/dev
main/test
minimum -> solid -> maximum
Before editing or executing:
active_baseline_metric_contract_json, read that JSON file before planning commands or comparisonsactive_baseline_metric_contract_json as the default authoritative baseline comparison contract unless you record a concrete reason to override itIf a repeated failure pattern already exists, apply the mitigation first and record that choice.
Also confirm before comparison work:
active_baseline_metric_contract_json when that file is availableactive_baseline_metric_contract_json; extra metrics are allowed, but missing required metrics are notmain/test and superiority is likely to be claimed, define the significance-testing plan before execution rather than after seeing the numbersResult/metric.md was used during the run, treat it as optional scratch memory only and reconcile it against the final submitted metrics before cs_artifact_record_main_experiment(...)Before you begin a substantial run, send a concise threaded cs_artifact_record payload={'kind': 'progress'}, ...) update naming:
Switch from ordinary execution mode into diagnosis mode when any of the following becomes true:
In diagnosis mode:
The normal experiment workspace is the current active idea worktree returned by artifact.submit_idea(...).
artifact.submit_idea(mode='create', lineage_intent='continue_line'|'branch_alternative', ...) instead of silently mutating the old nodecs_artifact_record_main_experiment(...) before moving to analysis or writingcontinue_line child branch or branch_alternative sibling-like branchcs_artifact_record_main_experiment(...), if external messaging milestone media is enabled and the metrics are stable enough to summarize honestly, prefer one concise summary PNG over multiple attachmentsImplementation rules:
PLAN.md instead of repeatedly improvising a new method after each small observationPrefer to complete one experiment cleanly before expanding to the next, unless parallel execution is explicitly justified and isolated. For substantial experiment packages, the default is one experiment at a time, with each one reaching a recoverable recorded state before the next begins.
Retry-delta discipline:
Run with auditable commands and durable outputs.
Execution rules:
cs_bash_exec instead of ephemeral shell invocationsactive_baseline_metric_contract_json silently when that file existsYou may do a quick sanity run first, but if the stage goal is a real experiment you must continue to the real evaluation unless the run is blocked and recorded.
Pilot-before-scale rule:
Incremental-recording rule:
CHECKLIST.md alongside those durable notes so the current execution frontier is obvious without replaying the whole logLast-known-good rule:
For commands that may run longer than a few minutes:
cs_bash_exec mode='detach', ...) and normally leave timeout_seconds unset for that long runcs_bash_exec mode='read', id=...) returns the full rendered log when it is 2000 lines or fewer; for longer logs it returns the first 500 lines plus the last 1500 lines and a hint to inspect omitted sections with start and tailcs_bash_exec mode='read', id=..., start=..., tail=...)cs_bash_exec mode='list') and cs_bash_exec mode='read', id=..., tail_limit=..., order='desc') to monitor or revisit managed commands while focusing on the newest evidence firstcs_bash_exec mode='read', id=..., after_seq=last_seen_seq, tail_limit=..., order='asc') so later checks only fetch new evidencecs_bash_exec mode='history')comment such as {stage, goal, action, expected_signal, next_check}silent_seconds, progress_age_seconds, signal_age_seconds, and watchdog_overdue from cs_bash_exec mode='list'|'read', ...) as your default watchdog signals60s, then inspect logs120s, then inspect logs300s, then inspect logs600s, then inspect logs1800s, then inspect logs1800s while the run is still activecs_bash_exec command='sleep 60', mode='await', timeout_seconds=70) or cs_bash_exec mode='await', id=..., timeout_seconds=...) between checkscs_bash_exec command='sleep N', mode='await', timeout_seconds=N+buffer, ...)timeout_seconds exactly equal to Ncs_bash_exec mode='await', id=..., timeout_seconds=...) instead of starting a new sleep commandcs_artifact_record payload={'kind': 'progress'}, ...) when the user-visible state, frontier, blocker status, or ETA materially changedcs_bash_exec mode='kill', id=..., wait=true, timeout_seconds=...); if it must die immediately, add force=true, record the reason, fix the issue, and relaunch cleanlyAlways preserve the managed cs_bash_exec log and export it into the experiment artifact directory when the run artifact is written.
Long loops should emit structured progress markers rather than noisy raw progress bars.
eta in seconds and next_reply_at or next_check_at so web/removed terminal UI can show the next expected updateIf you control the code, prefer a throttled tqdm-style progress reporter for the run itself and pair it with concise structured __CS_PROGRESS__ lines when feasible so monitoring remains machine-readable.
After the run, verify:
Create a durable claim-validation record that maps:
supportedrefutedinconclusiveAlso verify baseline comparability before claiming deltas:
Every meaningful main run must be recorded through cs_artifact_record_main_experiment(...).
That call is responsible for writing:
experiments/main/<run_id>/RUN.mdexperiments/main/<run_id>/RESULT.jsonrun artifact payloadcs_artifact_record_main_experiment(...) should include at least:
run_idmetrics_summarymetric_rows when availableevaluation_summary with exactly these six fields:
takeawayclaim_updatebaseline_relationcomparabilityfailure_modenext_actionUse evaluation_summary as the short structured judgment layer on top of the longer narrative fields:
takeaway: one sentence the next reader can reuse directlyclaim_update: strengthens, weakens, narrows, or neutralbaseline_relation: better, worse, mixed, or not_comparablecomparability: high, medium, or lowfailure_mode: none, implementation, evaluation, environment, or directionnext_action: the immediate route such as continue, revise_idea, analysis_campaign, write, or stopAfter cs_artifact_record_main_experiment(...) succeeds, do not assume the same branch should absorb the next round by default.
Interpret the measured result first, then either:
Use artifact.create_analysis_campaign(...) only when the extra slices have clear academic or claim-level value relative to their resource cost.
If the main need is simply to continue optimization from a measured result, prefer a new durable child idea branch instead of an expensive analysis package by reflex.
If the extra work should happen on an older durable branch rather than the current head, first switch the runtime back there with a Codex-controlled branch activation action(...), then launch the analysis campaign from that activated workspace.
When cs_artifact_record_main_experiment(...) succeeds, send a richer threaded cs_artifact_record payload={'kind': 'milestone'}, ...) update rather than a generic one-line progress ping.
Lead that milestone with a concise 1-2 sentence outcome summary before expanding into more detail.
That milestone should state:
Do not treat a main run as durably complete until cs_artifact_record_main_experiment(...) succeeds.
Recommended per-run documentation fields:
These seven fields should be progressively filled as the run advances, not only at final packaging time.
RUN.md should make it easy for later stages to answer:
When the run is analysis-heavy or meant to fill a writing evidence gap, prefer a structured summary with:
Recording rules:
The experiment stage should normally end with one of:
Do not let the stage end without an explicit next direction. If analysis is selected, record why the expected information gain is strong enough to justify the added compute, time, or annotation budget.
A credible main run should satisfy:
If the result is confounded, say so directly.
Before marking the run complete, verify all of the following:
active_baseline_metric_contract_json when that file existsIf these checks fail, record the run as partial or blocked rather than pretending it is complete.
Stage-start requirement:
cs_memory_search(scope='quest', limit=5)cs_memory_search(...) before a new run, retry, or material execution changeidea_id, branch, and run_id; do not casually reuse memory from another idea line unless you are explicitly comparing linesWrite to memory only when the lesson is reusable, such as:
The canonical record of the run itself belongs in artifact, not only in memory.
Preferred memory usage:
ideas:
decisions:
episodes:
knowledge:
knowledge:
templates:
Use tags to refine retrieval when helpful, for example:
stage:experimenttype:failure-patterntype:metric-contracttype:claim-validationtopic:<mechanism>When calling cs_memory_write(...), pass tags as an array like ["stage:experiment", "type:failure-pattern", "topic:<mechanism>"], not as one comma-joined string.
Recommended read timing:
ideas, decisions, and relevant knowledgeepisodes firstdecisionsepisodes and stable debugging knowledgeStage-end requirement:
cs_memory_write(...) must state whether the outcome was success, partial, or failurecs_memory_write(...) should also include the current idea_id, branch, and run_id so later retrieval does not mix different experiment linesTypical artifact sequence:
cs_artifact_record_main_experiment(...) at main-run completionPreferred artifact choices:
progress for long-running execution updatescs_artifact_record_main_experiment(...) for each meaningful completed main experimentrun for analysis slice records when cs_artifact_record_analysis_slice(...) writes themreport for:
milestone when a major stage checkpoint is reacheddecision for:
approval when an explicit user approval is captured for an expensive or risky run changeUse backend automatic checkpointing(...) when code evolution is meaningful and should be preserved in Git.
After a meaningful experiment checkpoint or completion, emit cs_artifact_record payload={'kind': 'progress'} | 'milestone', ...) so the user sees the concrete result and next step.
A failed main run is still useful if it is explained well.
Record:
Prefer a primary failure type such as:
data_contract_mismatchresource_exhaustednumeric_instabilityimplementation_bugevaluation_pipeline_failureexternal_dependency_blockeddirection_underperformingAlso classify the broader failure layer when possible:
Do not collapse these into one bucket. A direction should only be treated as failing when repeated, well-instrumented evidence still points to underperformance after implementation, evaluation, and environment explanations have been checked.
Blocked experiment states commonly include:
When results are suspicious, such as identical to baseline, implausibly perfect, or inconsistent across repeats, diagnose systematically:
Default diagnosis loop:
2-4 concrete hypothesesExit the experiment stage once one of the following is durably true:
analysis-campaign, write, another experiment, or resetWrite CodexScientist-aware implementation, experiment-roadmap, code-only, and formal-command plans with TDD and durable quest outputs.
Use when a quest needs one or more follow-up runs such as ablations, robustness checks, error analysis, or failure analysis after a main experiment.
Use when a quest needs to attach, import, reproduce, repair, verify, compare, or publish a baseline and its metrics.
Use when the quest needs an explicit go, stop, branch, reuse-baseline, write, finalize, reset, or user-decision transition with reasons and evidence.
Execute CodexScientist project experiment command documents with durable cs_* logging, manifest validation, and cautious baseline-gate handling.
Use when the quest is ready to consolidate final claims, limitations, recommendations, summary state, and graph exports before stopping or archiving.