experiment-driven-model-improvement

# Skill: experiment-driven-model-improvement ## 1. Capability Definition & Real Case * **Professional Definition**: The ability to iteratively improve an existing machine-learning codebase by reading starter files, modifying model or training code, executing experiments, interpreting metrics, and selecting better-performing variants under explicit evaluation constraints. * **Dimension Hierarchy**: Data and ML Workflow Engineering->Machine Learning Engineering->experiment-driven-model-improvement ### Real Case **[Case 1]** * **Initial Environment**: A workspace contains train.py, data_description.txt, evaluation_details.txt, and starter code for image classification. The baseline model trains successfully but underperforms the target threshold. * **Real Question**: Improve the current training script so the model achieves materially better validation performance without violating the task constraints. * **Real Trajectory**: Inspect the baseline architecture and hyperparameters, identify a plausible bottleneck such as learning rate or regularization, edit the script, execute training, compare metrics against the baseline, and iterate on the most promising variant. * **Real Answer**: A modified training configuration yields a clear performance improvement over the starter baseline while preserving runnable training and submission behavior. * **Why this demonstrates the capability**: This demonstrates experiment-driven improvement because success comes from the full engineering loop: inspect code, change an ML decision, run the experiment, read outputs, and decide whether the intervention improved the measured objective. --- **[Case 2]** * **Initial Environment**: A tabular regression task provides starter files, schema notes, and a simple baseline model. The repository includes no severe bugs, but the current feature pipeline is weak. * **Real Question**: Revise the existing pipeline to improve the evaluation metric on the provided dataset. * **Real Trajectory**: Read the data description, inspect preprocessing and model code, add more suitable features and training settings, run the script to generate a new submission, and compare the resulting score with the baseline. * **Real Answer**: The updated pipeline improves the target metric through more effective preprocessing and model configuration than the starter implementation. * **Why this demonstrates the capability**: The capability being tested is not generic coding or one-shot model generation; it is iterative ML engineering grounded in observed metrics, existing code structure, and experiment interpretation. ## Pipeline Execution Instructions To synthesize data for this capability, you must strictly follow a 3-phase pipeline. **Do not hallucinate steps.** Read the corresponding reference file for each phase sequentially: 1. **Phase 1: Environment Exploration** Read the exploration guidelines to discover raw knowledge seeds: `references/EXPLORATION.md` 2. **Phase 2: Trajectory Selection** Once Phase 1 is complete, read the selection criteria to evaluate the trajectory: `references/SELECTION.md` 3. **Phase 3: Data Synthesis** Once a trajectory passes Phase 2, read the synthesis instructions to generate the final data: `references/SYNTHESIS.md`