context-engineering-hillclimb

> **Load [`context-engineering-workflow`](../context-engineering-workflow/SKILL.md) first** for shared terminology, lifecycle overview, and safety protocol. # Skill: Automated Hill-Climbing ## Goal Given a golden dataset and (optionally) a base context, autonomously produce a high-quality ContextSet by iterating evaluate → analyze → mutate → re-upload until convergence. The user supplies inputs once and receives the final high-scoring ContextSet; no per-iteration approval is required. ## Prerequisites - A working DB connection — Toolbox MCP tools (`<source>-list-schemas`, `<source>-execute-sql`) must be visible to the agent throughout the run. If missing or unreachable at any point, stop and route through `context-engineering-init`; do not work around it (no bash `uvx toolbox-server invoke` fallback). - A golden evaluation dataset split by `context-engineering-dataset-generation`, stored **inside the experiment** (`.context-engineering/experiments/<experiment_name>/dataset/golden.json`, `dataset/splits/hillclimb.json`, `dataset/splits/holdout.json`) and recorded in `experiment_state.md` — the loop evaluates on `splits/hillclimb.json` only; `splits/holdout.json` is never read until the loop has converged (see the Holdout Evaluation phase in `context-engineering-workflow`). - ADC configured and the Gemini Data Analytics + Dataplex APIs enabled on the project (see `context-engineering-init` for preflight). - A starting context — none, a local file, or an existing Context Set resource name. Entry flow spells out the handling per case. - An experiment `experiment_state.md` written by `context-engineering-init` at `.context-engineering/experiments/<experiment_name>/hillclimb/experiment_state.md`. It holds the context set identity as resource names — `Context set id`, `Final Context Set resource` (the name the **final** context set lives under), the `Draft Context Set resource` bullet that decides the working copy (none for a fresh name, `<context_set_id>_draft` when the name already existed), and the optional `Seed Context Set resource` — plus `## Active Database`. The workspace inputs — Context Set `Project`, Context Set `Location` (where the Context Set is stored), `Enrichment sources`, `Auto-approve uploads` — live in `.context-engineering/workspace_state.md` (written by init), and `experiment_state.md` may carry a `(this experiment)` override for the last two; see *Inputs & resolution rules* in [ctxset-mcp-protocol.md](../context-engineering-workflow/references/ctxset-mcp-protocol.md). If either file is missing, route to `context-engineering-init` first. See the Context Set MCP Protocol in [ctxset-mcp-protocol.md](../context-engineering-workflow/references/ctxset-mcp-protocol.md). - (Optional) Workspace root directory. Default: `.context-engineering/experiments/<experiment_name>/` at cwd. ## Guidance Load [references/layout.md](references/layout.md) before any workspace interaction — it describes the internal iteration layout. ### How the Context Set server is used in this loop Local files (`vN/context_set_vN.json`) are the source of truth. The Context Set server holds **one working copy** (`<working_copy>`) for the whole run (read `Draft Context Set resource` from `experiment_state.md`; see [references/finalize.md](references/finalize.md) and [ctxset-mcp-protocol.md](../context-engineering-workflow/references/ctxset-mcp-protocol.md) for the full table): - **Fresh case** (`Draft Context Set resource` is `(none — new context set; iterations upload to the final name)`): `<working_copy>` = the bare `<context_set_id>`. Pass `databases` derived from `## Active Database` **Source Name** + `.context-engineering/tools.yaml`. No `_draft`, nothing deleted. - **Existing case** (`Draft Context Set resource` is `…/<context_set_id>_draft`): `<working_copy>` = `<context_set_id>_draft`. **Omit `databases`** on `_draft` uploads so temporary drafts do not appear in the Cloud Console DB Studio UI; the existing `<context_set_id>` is never written during the loop and is only replaced at Finalize after `PASS` on holdout + explicit user approval, followed by deleting `_draft`. Every `upload_context_set` and `delete_context_set` call parses `(projectId, location, contextSetId)` from the resource names in `experiment_state.md` (never `workspace_state.md`), passes the exact file contents in `contextPayload`, and polls `get_operation(projectId, location, operationId)` ($\ge 5\text{ s}$ between polls, 5-minute ceiling) until `done: true` before continuing. **Upload approval — two tiers for iteration uploads.** The effective `Auto-approve uploads` is: `Auto-approve uploads (this experiment)` in `experiment_state.md` if present → else `Auto-approve uploads` in `.context-engineering/workspace_state.md` → else `false`. Re-read it on every entry; never modify `Auto-approve uploads` or any resource identity bullet (`Final Context Set resource`, `Draft Context Set resource`, `Seed Context Set resource`), and verify `(projectId, location, contextSetId)` against the allow-listed identities in `experiment_state.md` before every upload or delete. - `true` (only when the user answered init's approval question with "automatic"; never inferred from a one-off "upload it" instruction) → every working-copy upload proceeds without asking. - `false`, or neither bullet exists → before each working-copy upload, pause and say: *"Draft iteration `vN` is ready in `vN/context_set_vN.json`. Please review and approve it, and I'll upload this draft to be scored."* Wait for approval; if the user edits the file, re-run `validate_context_set` before uploading. Never assume `true`. Replacing an **existing** context set (or syncing back to a server seed) at Finalize is always gated by the mandatory HITL in [references/finalize.md](references/finalize.md), regardless of tier. ### Entry flow 1. **Locate the experiment.** `<experiment_root>` = `.context-engineering/experiments/<experiment_name>/` holds `dataset/` and **`hillclimb/`**; `<hillclimb_root>` = `<experiment_root>/hillclimb/` holds `experiment_state.md`, every `vN/`, `holdout_eval/` and `final_evaluation_report.md`. **All relative paths in this skill (`vN/…`, `holdout_eval/…`) are relative to `<hillclimb_root>`**; the DB connection is the `tools.yaml` source named under `## Active Database`. If `<hillclimb_root>/experiment_state.md` is missing → route to `context-engineering-init`. If `experiment_state.md` has `## Final:` → the experiment is **finished and read-only**: do not iterate; report the outcome and, unless the verdict was PASS, re-print the *Next experiment hand-off*; route to `context-engineering-init` in clone mode if the user wants to continue. If `experiment_state.md` has an `## Iteration Log` → **resume** from the recorded iteration (see Resume rules in [references/layout.md](references/layout.md)). Otherwise → **fresh start**. 2. **Read the experiment identity from `experiment_state.md`** (`Context set id`, `Final Context Set resource`, `Draft Context Set resource`, `Seed Context Set resource`, `Hillclimb dataset`) and parse `projectId` / `location` from `Final Context Set resource`. Read the effective `Auto-approve uploads` and `Enrichment sources` per *Inputs & resolution rules* (`experiment_state.md` `(this experiment)` override → `.context-engineering/workspace_state.md`). Do not re-ask for any of them; if a bullet is missing, route to `context-engineering-init`. Set **`<working_copy>`** = `<context_set_id>` (fresh case) or `<context_set_id>_draft` (existing case). 3. **Fresh start — seed v0.** Produce `v0/context_set_v0.json` per the base-context case: - **None** → invoke `context-engineering-bootstrap` to generate `v0/context_set_v0.json` (no upload from bootstrap; this loop handles uploads). Bootstrap reads the DB source from `experiment_state.md ## Active Database` and the effective enrichment sources (step 2). - **Local file** → copy it to `v0/context_set_v0.json` and run `validate_context_set` on `v0/context_set_v0.json` (if invalid, fix each issue via `mutate_context_set` and re-validate until clean before uploading). - **Existing Context Set resource name** (`Seed Context Set resource` in `experiment_state.md`) → parse `(projectId, location, contextSetId)` per the Resource naming rule (accept any collection segment), call `get_context_set(projectId, location, contextSetId)` (if `NOT_FOUND`, stop and tell the user that `<contextSetId>` does not exist in `<location>` — do not probe other locations), parse `payload`, and write it to `v0/context_set_v0.json`. Never write to the seed during the loop. Then baseline-evaluate `v0`: upload `v0/context_set_v0.json` to `<working_copy>` (include derived `databases` in the fresh case; omit `databases` on `<context_set_id>_draft` in the existing case and tell the user once that iterations run on `<draft>` while `<Final Context Set resource>` stays untouched until Finalize) (approval tier applies) → poll gate → evaluate into `v0/eval/` on `Hillclimb dataset` (`<experiment_root>/dataset/splits/hillclimb.json`) → record the baseline score. Write `### v0` to `experiment_state.md` and print in chat: `🔄 Baseline (v0) complete: <passed>/<total> (<accuracy>%) — <Next: starting v1 to address <top failure category> | Tuning target reached: running holdout test...>`. Iteration loop starts at `v1`. ### Per-iteration loop (`vN`) 1. **Prepare `vN/`**: append the `## In-Progress: vN` marker to `experiment_state.md`, create the iteration directory, and seed `context_set_vN.json` by copying `v(N-1)/context_set_v(N-1).json` from disk. Local files are the only durable record, so there is no remote fallback: if the local file is missing (crash), copy the most recent `vK/context_set_vK.json` that does exist; if none exists, re-run the entry-flow seed. 2. **Analyze the previous evaluation**: read `v(N-1)/eval/` via `read_evaluation_result`; cluster failures by category; write findings + reasoning + planned mutations to `analysis_vN.md`. - **Pagination**: The tool returns a batch of failure cases (default limit 10). If there are many failures, iterate by calling the tool with increasing `offset` (0, 10, 20...) until all failed queries are analyzed. - **Use `pipeline_debug_info`**: If a failure case's Additional Output contains it, use this generation trace (showing which context the API retrieved and used) to ground the root cause (e.g., distinguishing a missing template match vs. a bad column reference) rather than guessing. 3. **Mutate**: plan mutations from the analysis (prefer fewer general items — a facet often beats many templates). Author new items via `context-engineering-generation-guide`. Validate column references via `<source>-list-schemas` and generated SQL via `<source>-execute-sql`: execute **only read-only `SELECT` / `WITH` (or `GRAPH ... MATCH`) queries** (never DML/DDL), always add a small **`LIMIT`** (e.g., `LIMIT 10`) and a **statement timeout** where supported, wrap validation in a **read-only transaction** (`SET TRANSACTION READ ONLY` on Postgres; on Spanner, `readOnly: true` is configured on the `spanner-execute-sql` tool in `tools.yaml`), and use **`dryRun` / `maximumBytesBilled`** on BigQuery to bound scan cost. Apply via `mutate_context_set` to `context_set_vN.json`, then run `validate_context_set` on `vN/context_set_vN.json` (if invalid, fix each issue via `mutate_context_set` and re-validate until clean before uploading). 4. **Upload the working copy** (approval tier applies): read `vN/context_set_vN.json` and call `upload_context_set(projectId=<parsed from Final Context Set resource>, location=<parsed from Final Context Set resource>, contextSetId=<working_copy>, contextPayload=<exact file contents>, databases=<derived from the Active Database source + tools.yaml in the fresh case; omit databases when <working_copy> is <context_set_id>_draft>)` → poll gate until `done: true`. This overwrites the previous iteration's copy. Record the operation name in `experiment_state.md`. 5. **Evaluate**: invoke `context-engineering-evaluate` with `context_set_id=<the working-copy resource>`, the `Hillclimb dataset` path from `experiment_state.md` (`<experiment_root>/dataset/splits/hillclimb.json`), and `output_dir=<hillclimb_root>/vN/eval/`. Capture `job_id`, `passed`, `total` and overall score. 6. **Update `experiment_state.md` and print progress**: replace the `## In-Progress: vN` marker with the final `### vN` entry (local file, eval report path, analysis path, score, upload operation name). Print a concise one-line progress update in chat so the user can follow the run: `🔄 Iteration vN complete: <passed>/<total> (<accuracy>%, <±X% vs best so far>) — <Next: starting v(N+1) to address <top failure category> | Converged (<reason>): running holdout test...>`. 7. **Check the stopping conditions** (see below). If none fired, continue to `v(N+1)`. ### Stopping conditions and finalization The stopping rules are **internal constants of this skill** — they are not asked of the user by init and are **not recorded in `experiment_state.md`** (only the outcome is). After every evaluation, check them in this order: | Condition | Constant | Fires when | `## Converged:` reason | | :--- | :--- | :--- | :--- | | Tuning target | `TUNING_TARGET = 1.0` | `passed / total >= TUNING_TARGET` (i.e. zero failures) | `Tuning target <t> reached.` | | Plateau | `PLATEAU_K = 3` | the last `PLATEAU_K` iterations all failed to beat the best score so far | `Plateau reached over <k> consecutive mutations.` | | Iteration cap | `MAX_ITERATIONS = 10` | `MAX_ITERATIONS` iterations `v1..vN` have been evaluated | `Maximum iterations (<n>) reached.` | Power users (and smoke tests) may override a constant **explicitly in their prompt** for the current session ("max iterations 3"); honour it, say so once, but do not write it to `experiment_state.md` — a later resume falls back to the constants unless the user restates the override. The `## Converged:` reason already carries the effective value. When one fires, append `## Converged: <reason>` to `experiment_state.md`, **load [references/finalize.md](references/finalize.md)**, and execute **Finalize** immediately, in the same turn, without asking. (The only pause inside Finalize is the replacement/seed-sync HITL when an existing server Context Set is involved.) The user can also explicitly ask to stop at any time; the current iteration completes cleanly, `## User Stop` is appended to `experiment_state.md`, and all iteration files are preserved. A user stop does **not** finalize and does **not** run the holdout test — a later resume offers the choice. **Finalize summary** (full protocol in [references/finalize.md](references/finalize.md) — **must load before executing Finalize**): 1. Pick the best-scoring iteration `K` from the Iteration Log (ties → earliest) and append `## Finalizing: vK` to `experiment_state.md`. 2. Re-upload `vK/context_set_vK.json` to `<working_copy>` if `K ≠ N` (or unconditionally on resume) and poll `get_operation` to `done: true`. 3. Load [`../context-engineering-workflow/references/holdout-evaluation.md`](../context-engineering-workflow/references/holdout-evaluation.md) and run the single read-only Holdout Evaluation & Generalization Reporting Phase on `<working_copy>`. 4. Finish per case per [references/finalize.md](references/finalize.md) (fresh case vs. existing case `_draft` replacement HITL, preserving existing `databases` associations, and deleting `<context_set_id>_draft`), write `## Final:`, and print the On-Screen Card (plus the Next-Experiment Hand-off card if the verdict is not `PASS`). ## Rules - The workspace is internal state; user-visible output is the final Context Set resource name and score. Do not surface intermediate iteration files as user deliverables.