context-engineering-hillclimb

name	context-engineering-hillclimb
description	Guides the agent to perform hill-climbing iterations to improve a ContextSet based on Evalbench evaluation results.

Load the context-engineering-workflow skill first. It holds the shared context this phase depends on: workspace layout, state file conventions, phase order, and safety protocol. Do not proceed with this phase without reading it.

Phase: Optimization & Hill-Climbing

Goal

Analyze evaluation failures to perform a Gap Analysis and apply targeted context mutations (refinements) to iteratively improve translation accuracy.

Workflow

Follow these steps exactly in order:

1. Setup & Loop

1. Validation:
   • Check if autoctx/experiments/ directory and autoctx/state.md exist. If missing, warn the user that the workspace might not be initialized (suggest running the Setup & Connection phase of this skill first).
   • Once an experiment is selected, verify it contains an eval_reports/ folder. If missing, suggest running the Evaluation workflow first.
2. Identify Experiment:
   • Read the local autoctx/state.md to identify the active experiment.
   • If not found, ask the user to select an experiment folder from autoctx/experiments/.
3. Determine Loop Version:
   • Scan the autoctx/experiments/<experiment_name>/hillclimb/ folder for files matching improved_context_v*.json.
   • Determine the loop version vN by finding the maximum N and using N+1. If the folder is empty, start at v1.
4. Locate Base Context:
   • For v1:
     • Check autoctx/state.md to see if a specific base context path was recorded for this experiment (e.g., during the Evaluation setup for user-provided contexts).
     • If not found in state.md, default to the baseline generated by Bootstrap in the experiment folder.
     • If still not found, ask the user for the absolute path to their base context file and record it as the Base Context in autoctx/state.md.
   • For vN (where N > 1), the base context is improved_context_v(N-1).json.
   • Verify the base context file exists. If missing, STOP and ask the user for the correct path.

---

2. Phase 1: Gap Analysis

1. Validation:

   • Determine the target evaluation run folder under eval_reports/. If multiple folders exist, find the most recent one by modified time. Prefer the latest run by default, but list other available runs as well (peeking into their summary.csv or configs.csv to show timestamps/metrics for visual context). Ask the user to confirm the selection.
   • Verify that the selected eval_reports/<job_id_folder>/ contains expected files (e.g., scores.csv, summary.csv). If missing or empty, STOP and inform the user.

2. Read Evaluation Results: Use the read_evaluation_result MCP tool passing the path to eval_reports/<job_id_folder>/.

3. Generate Gap Analysis Report (Batched):

   • The tool returns a summary and a batch of failure cases (default limit 10).
   • Iterate through the failure cases by calling the tool with increasing offset (0, 10, 20, ...) until all failed queries are analyzed.
   • First Batch (offset=0): Initialize the report file with the # Gap Analysis Report - vN header and ## Summary section, followed by the analysis of the first batch under ## Failed Queries Detail.
   • Subsequent Batches: Call the tool with the next offset, analyze the new failures, and append them to the ## Failed Queries Detail section.

   Use the following structure for the report:

   # Gap Analysis Report - vN
   
   ## Summary
   - **Total Queries**: 10
   - **Passed**: 7
   - **Failed**: 3
   - **Pass Rate**: 70%
   
   ## Failed Queries Detail
   
   ### Query 1: "How many users registered in 2023?"
   - **Error Category**: `[FilterError]`
   - : 
    :  (Missing filter)
    : The LLM did not know about the  column or how to filter by year for this entity.
    : Add a facet for "Users by Year".
   
   
    : 
    : 
    : 
    : Missing ordering instruction in context.
    : Update the template for "Product Sales" to include ordering.
   
   
    : 
    :  (Syntax error  in golden SQL)
    : 
    : Invalid syntax in golden dataset.
    : None. Flag to user to fix the evaluation dataset.
   

---

3. Phase 2: Context Mutation

Refer to context-generation-guide for how to edit a ContextSet.

1. Validation: Verify that gap_analysis_vN.md exists and contains findings. Verify the base ContextSet file exists. If missing, STOP and inform the user.
2. Analyze Gap Report & Determine Fixing Strategy:
   • Read gap_analysis_vN.md to identify what needs to be fixed.
   • Fixing Strategy Guidelines:
     • Conciseness: Try to use less context to cover more scenarios. Avoid adding redundant or hyper-specific templates for every single edge case.
     • Generalizability: Prefer solutions that generalize well (e.g., use a facet for a column definition rather than a specific template for every query using that column).
     • Supported Types: Support mutations for template, facet, and value_search types.
3. Apply Mutations:
   • Copy the Base Context: Copy the base ContextSet file to the new destination: autoctx/experiments/<experiment_name>/hillclimb/improved_context_vN.json.
   • Generate New Items: For any new context items identified in the fixing strategy (for "add" operations):
     • Invoke the context-generation-guide skill to produce the final parameterized items.
     • Provide the identified candidates to that skill.
     • That skill will handle phrase extraction, parameterization, and constructing the valid JSON structure.
   • Validate New Items:
     • Templates: Run generated SQL examples via <source>-execute-sql (use dummy values for placeholders) to verify syntax.
     • Others: Cross-check table/column references against the schema via <source>-list-schemas.
   • Apply Mutations: Call the mutate_context_set MCP tool passing the new file path as file_path and mutations as mutations_json to mutate the context set.
4. Validate: Call validate_context_set on improved_context_vN.json. If invalid, fix each issue via mutate_context_set and re-validate until clean. Stop after two failed attempts and surface remaining issues to the user.

---

4. Validation & Upload Advice

1. Summarize Improvements: Tell the user what was changed (e.g., added 2 facets, updated 1 template).
2. Upload Instructions:
   • Read Database Details: Read autoctx/tools.yaml (or db_config.yaml) to fetch the specific project, location, and instance/cluster details for the active database.
   • Generate URL: Call the generate_upload_url tool passing the extracted values to provide the direct console link to the user.
   • Present the local file path to improved_context_vN.json and the generated console link together in a single clear message.
3. Instruct Next Step Evaluation:
   • Instruct the user to run evaluation using the evaluating workflow on this new ContextSet to see if metrics improve. This will start Loop N+1.

---

Output

Upon successful completion, the workspace must contain:

• autoctx/experiments/<experiment_name>/hillclimb/gap_analysis_vN.md
• autoctx/experiments/<experiment_name>/hillclimb/improved_context_vN.json
• Updated autoctx/state.md summarizing the run loop.

---

Logging State Example (autoctx/state.md)

When updating autoctx/state.md, please append or update the Hill-Climbing Run Log section:

# Context Authoring Experiment State Tracking

## Active Experiment: my-exp-1

## Hill-Climbing Run Log

### Loop: v1
- **Base Context**: `baseline_context.json`
- **Eval Report Path**: `autoctx/experiments/my-exp-1/eval_reports/<job_id_uuid>/` (containing `configs.csv`, `evals.csv`, etc.)
- **Gap Analysis**: `autoctx/experiments/my-exp-1/hillclimb/gap_analysis_v1.md`
- **Mutated Context**: `autoctx/experiments/my-exp-1/hillclimb/improved_context_v1.json`

[!IMPORTANT] Tool Modification Rule: Always use the mutate_context_set tool for all ContextSet changes. Pass mutation payloads directly to the tool — it handles all file I/O internally. Do not read the target context set file beforehand.