experiment-reproduction-and-result-verification

# Skill: experiment-reproduction-and-result-verification ## 1. Capability Definition & Real Case * **Professional Definition**: The ability to run, monitor, validate, and assess whether an implemented research codebase actually reproduces the empirical results or intermediate execution outcomes required by a benchmark or rubric. * **Dimension Hierarchy**: Research Reproduction Engineering->Reproduction and Evaluation->experiment-reproduction-and-result-verification ### Real Case **[Case 1]** * **Initial Environment**: A repository includes implementation code, a reproduce.sh entrypoint, documentation, and a rubric describing what outcomes count as successful replication. Running the script generates logs, tables, and plots. * **Real Question**: Execute the reproduction pipeline and determine whether the target experimental results have been successfully reproduced. * **Real Trajectory**: Run reproduce.sh in a clean environment, inspect reproduce.log and generated outputs, compare observed artifacts against the required result criteria, and record which parts fully match, partially match, or fail. * **Real Answer**: The reproduction run achieves some required outputs but misses others, yielding a structured assessment of execution success and result fidelity. * **Why this demonstrates the capability**: This capability centers on verification after implementation. The challenge is interpreting empirical outputs against explicit criteria, not just producing code or running commands blindly. --- **[Case 2]** * **Initial Environment**: A partially complete research submission contains code development artifacts and a rubric with code, execution, and result-match requirements. The benchmark awards partial credit based on which requirements are satisfied. * **Real Question**: Assess the submission by checking which implementation, execution, and result requirements are actually met after running the pipeline. * **Real Trajectory**: Read the rubric, identify leaf requirements, run the provided scripts, inspect logs and outputs, and connect observed evidence to each requirement type before scoring the submission. * **Real Answer**: The submission satisfies some code-development and execution requirements but only a subset of result-match requirements, leading to a partial replication score. * **Why this demonstrates the capability**: The task demonstrates experiment reproduction and result verification because it requires evidence-based judgment over code, execution artifacts, and empirical outputs. The agent must distinguish having code from having runnable code, and runnable code from actually matched results. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`