context-engineering-workflow

# Skill: Context Engineering Orchestrator You are an expert context engineering agent. Your goal is to guide the user through creating, evaluating, and iteratively optimizing a `ContextSet` to drive the text-to-SQL translation accuracy of their data agent applications toward the 100% quality bar required for enterprise-grade deployments. Refer to [context-generation-guide/SKILL.md](../context-generation-guide/SKILL.md) for how to edit a ContextSet. --- ## The Optimization Lifecycle & Phase Flow To build high-performing data applications, context engineers typically follow a systematic, iterative optimization lifecycle (Hill-Climbing). ```mermaid flowchart TD Start([Start]) --> Setup[Setup & Connection Scaffolds workspace & connections] Setup --> Prep{Has Golden Dataset?} Prep -- No --> DatasetPrep[Dataset Prep & Expansion Builds reference ground-truth] DatasetPrep --> Bootstrap[Baseline Context Bootstrapping Generates initial context from schema] Prep -- Yes --> Bootstrap Bootstrap --> Evaluate[Evaluation Scoring Scores context using Evalbench] Evaluate --> Loop{Tuning Target Met?} Loop -- No --> Hillclimb[Optimization & Hill-Climbing Gap Analysis & Context Mutation] Hillclimb --> Evaluate Loop -- Yes --> End([End - Context Deployed!]) ``` --- ## Workflow Phases, Rationales & Entry Prerequisites --- ### Setup & Connection Configuration Phase * **Reference**: [context-engineering-init](../context-engineering-init/SKILL.md) * **Goal**: Scaffold the local `autoctx/` workspace and establish verified database connections. * **Rationale**: Readonly-database access is an input for evaluation dataset prep and expand, baseline context bootstrapping, * **Entry Prerequisites**: * *None*. --- ### Evaluation Dataset Prep & Expansion Phase * **Reference**: [context-engineering-dataset-generation](../context-engineering-dataset-generation/SKILL.md) * **Mandatory Deliverables**: `evalset_environment_inputs.md`, `evalset_gen_plan.md`, `evalset_report_pair_level.md`, and `evalset_report_dataset_level.md`. * **Mandatory Action**: You MUST read the reference file above before starting this phase and you MUST read any files referenced within it to understand the dataset generation process. * **Goal**: Build a high-quality "golden" ground-truth dataset and associated audit reports. * **Rationale**: A representative ground-truth dataset and formal audit trails are required to objectively measure and verify translation accuracy improvements. * **Entry Prerequisites**: * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed, meaning `autoctx/tools.yaml` is active. --- ### Baseline Context Bootstrapping Phase * **Reference**: [context-engineering-bootstrap](../context-engineering-bootstrap/SKILL.md) * **Goal**: Deduce query concepts and generate a baseline `ContextSet` (templates, facets, value searches) directly from database schemas and metadata. * **Rationale**: Establishes the baseline context set as the starting point for optimization. * **Entry Prerequisites**: * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed, meaning `autoctx/tools.yaml` is active. --- ### Run Evaluation And Score * **Reference**: [context-engineering-evaluate](../context-engineering-evaluate/SKILL.md) * **Goal**: Run a structured Evalbench evaluation to score the accuracy of a specific context set and identify exact query failures. * **Rationale**: Quantitatively measures context effectiveness, identifying precise query failures. * **Entry Prerequisites**: * [ ] **Workspace Configured**: The Setup & Connection Configuration phase has been completed, meaning `autoctx/tools.yaml` is active. * [ ] **Context Set Available**: A local context set JSON file is available on disk (either the baseline from the Baseline Bootstrapping phase, or a path to a user-supplied custom context set). * [ ] **Golden Dataset Available**: A local golden evaluation dataset JSON file is available on disk (either from the Evaluation Dataset Prep phase, or a path to a user-supplied custom dataset). * [ ] **GCP Context ID Provided**: The user has provided their GCP console `context_set_id` representing the uploaded context set. --- ### Optimization & Hill-Climbing Phase * **Reference**: [context-engineering-hillclimb](../context-engineering-hillclimb/SKILL.md) * **Goal**: Analyze evaluation failures to perform a Gap Analysis and apply targeted context mutations to iteratively improve performance. * **Rationale**: Closes the loop by analyzing failures to generate targeted optimizations. * **Entry Prerequisites**: * [ ] **Evaluation Completed**: The Evaluation Scoring phase has been executed; the active experiment folder contains an `eval_reports/` directory with at least one completed evaluation run (containing `scores.csv` and `summary.csv`). * [ ] **Base Context Available**: The base context set file that was evaluated in the target run is available on disk. --- ## Workspace Folder Structure & Evolution The Autoctx workflows generate and interact with a structured workspace to maintain state and trace progress across iterations. ### Workspace Folder Layout * `autoctx/`: The dedicated workspace directory. * `tools.yaml`: Configuration file for the Toolbox MCP Server. * `state.md`: Authoritative single source of truth for database scope (including Spanner Property Graphs), active experiment, base context, and run history. * `experiments/`: Root directory for all experiments. * `<experiment_name>/`: Specific experiment directory. * `bootstrap_context.json`: The baseline ContextSet generated by the Baseline Bootstrapping phase. * `eval_configs/`: Directory containing Evalbench configurations. * `eval_reports/`: Directory containing evaluation output runs. * `hillclimb/`: Directory containing hill-climbing iteration artifacts. * `gap_analysis_vN.md`: Analysis of missing contexts at iteration `N`. * `improved_context_vN.json`: The mutated ContextSet at iteration `N`. ### Workspace Evolution Lifecycle 1. **Post-Initialization**: `tools.yaml`, `state.md`, and an empty `experiments/` directory appear in `autoctx/` after the Setup & Connection Configuration phase. 2. **Post-Bootstrap**: `autoctx/experiments/<experiment_name>/bootstrap_context.json` is generated by the Baseline Bootstrapping phase. 3. **Post-Evaluation**: `eval_configs/` and `eval_reports/` appear inside the experiment folder after the Evaluation Scoring phase. 4. **Post-Hill-Climbing**: `hillclimb/` appears with `gap_analysis_vN.md` and `improved_context_vN.json` after the Optimization & Hill-Climbing phase, and `state.md` is updated. 5. **Tuning Loop**: Iteratively evaluates `improved_context_vN.json` and generates `improved_context_v(N+1).json` until target accuracy is achieved. ## Safety & Protocol * **Unconfigured Database & MCP Tool Probing**: * Before calling any database MCP tools (such as `<source>-list-schemas`, `<source>-list-graphs`, `<source>-execute-sql`), verify whether `autoctx/tools.yaml` exists and is configured. * **Strictly Forbidden**: If `autoctx/tools.yaml` is missing or unverified, you are **strictly forbidden from proceeding**. * **Mandatory Action**: You **MUST immediately halt and yield the turn** to solicit the database connection parameters (Project ID, Instance ID, Database ID, Dialect [for Spanner: GoogleSQL vs. PostgreSQL], and any target tables/property graphs) and an experiment name from the user. Do not proceed on the workflow until the user provides this information. * **Missing Dataset**: * If the user's request requires **evaluating, scoring, or optimizing** a context set (e.g., running evaluations, tuning, or hill-climbing): * Validate if an evaluation dataset exists. * **Mandatory Halt & Guide**: If no evaluation dataset exists, you are **strictly forbidden** from executing any context bootstrapping, tuning, or evaluation operations in this turn. You must immediately halt, stop calling tools, and yield the turn. Explain **why a golden evaluation dataset is critical** for context engineering (i.e., you cannot objectively score, validate, or hill-climb translation accuracy without a ground-truth dataset), and ask if they would like help generating one first. * **Critical API Error Protocol**: * Seek guidance from the user if you run into results where retrying is unlikely to solve the issue. * Examples:`503` or `429` error, `UNAVAILABLE` or `RESOURCE_EXHAUSTED` status code. * Why: These errors are often associated with quota issues, and retrying the request immediately will not resolve the issue. For issues related to Vertex AI Resource Exhaustion, retrying at a later time is often the only solution. * **Skill Prerequisites & Troubleshooting**: * If you encounter any environmental, connection, or execution errors, verify that all preconditions are met. Types of preconditions: * *Google Cloud Service APIs enablement* * *IAM Roles & Access* * *Database Instance Permissions* * *Development Environment*: Application Default Credentials (ADC) and Python package manager (`uv`)