# Context Engineering Hillclimb > Guides the agent to perform hill-climbing iterations to improve a ContextSet based on Evalbench evaluation results. - Skill: `googlecloudplatform/context-engineering-hillclimb` (Agent Skill) - Install (CLI): `npx skillmds@latest add googlecloudplatform/context-engineering-hillclimb` - Raw SKILL.md: https://api.skillmd.com/api/skills/googlecloudplatform/context-engineering-hillclimb/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: GoogleCloudPlatform (https://skillmd.com/u/googlecloudplatform) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/googlecloudplatform/context-engineering-hillclimb --- > **Load the `context-engineering-workflow` skill first.** It holds the shared context this phase depends on: workspace layout, state file conventions, phase order, and safety protocol. Do not proceed with this phase without reading it. # Phase: Optimization & Hill-Climbing ## Goal Analyze evaluation failures to perform a Gap Analysis and apply targeted context mutations (refinements) to iteratively improve translation accuracy. ## Workflow Follow these steps exactly in order: ### 1. Setup & Loop 1. **Validation**: - Check if `autoctx/experiments/` directory and `autoctx/state.md` exist. If missing, warn the user that the workspace might not be initialized (suggest running the Setup & Connection phase of this skill first). - Once an experiment is selected, verify it contains an `eval_reports/` folder. If missing, suggest running the Evaluation workflow first. 2. **Identify Experiment**: - Read the local `autoctx/state.md` to identify the active experiment. - If not found, ask the user to select an experiment folder from `autoctx/experiments/`. 3. **Determine Loop Version**: - Scan the `autoctx/experiments//hillclimb/` folder for files matching `improved_context_v*.json`. - Determine the loop version `vN` by finding the maximum `N` and using `N+1`. If the folder is empty, start at `v1`. 4. **Locate Base Context**: - For `v1`: - Check `autoctx/state.md` to see if a specific base context path was recorded for this experiment (e.g., during the Evaluation setup for user-provided contexts). - If not found in `state.md`, default to the baseline generated by Bootstrap in the experiment folder. - If still not found, ask the user for the absolute path to their base context file and record it as the Base Context in autoctx/state.md. - For `vN` (where N > 1), the base context is `improved_context_v(N-1).json`. - Verify the base context file exists. If missing, STOP and ask the user for the correct path. --- ### 2. Phase 1: Gap Analysis 1. **Validation**: - Determine the target evaluation run folder under `eval_reports/`. If multiple folders exist, find the most recent one by modified time. **Prefer the latest run by default**, but list other available runs as well (peeking into their `summary.csv` or `configs.csv` to show timestamps/metrics for visual context). Ask the user to confirm the selection. - Verify that the selected `eval_reports//` contains expected files (e.g., `scores.csv`, `summary.csv`). If missing or empty, STOP and inform the user. 2. **Read Evaluation Results**: Use the `read_evaluation_result` MCP tool passing the path to `eval_reports//`. 3. **Generate Gap Analysis Report (Batched)**: - The tool returns a summary and a batch of failure cases (default limit 10). - Iterate through the failure cases by calling the tool with increasing `offset` (0, 10, 20, ...) until all failed queries are analyzed. - **First Batch (offset=0)**: Initialize the report file with the `# Gap Analysis Report - vN` header and `## Summary` section, followed by the analysis of the first batch under `## Failed Queries Detail`. - **Subsequent Batches**: Call the tool with the next offset, analyze the new failures, and **append** them to the `## Failed Queries Detail` section. - **Use `pipeline_debug_info`**: If a failure case's **Additional Output** contains it, use this generation trace (showing which context the API retrieved and used) to ground the **Root Cause** and **Proposed Mutation** rather than guessing. For example, if the trace shows no template matched a question that should have been covered, treat the missing blueprint match as the root cause. Use the following structure for the report: ```markdown # Gap Analysis Report - vN ## Summary - **Total Queries**: 10 - **Passed**: 7 - **Failed**: 3 - **Pass Rate**: 70% ## Failed Queries Detail ### Query 1: "How many users registered in 2023?" - **Error Category**: `[FilterError]` - **Expected SQL**: `SELECT count(*) FROM users WHERE year = 2023` - **Actual SQL**: `SELECT count(*) FROM users` (Missing filter) - **Root Cause**: The LLM did not know about the `year` column or how to filter by year for this entity. - **Proposed Mutation**: Add a facet for "Users by Year". ### Query 2: "Show me top selling products" - **Error Category**: `[OrderingError]` - **Expected SQL**: `SELECT name FROM products ORDER BY sales DESC LIMIT 5` - **Actual SQL**: `SELECT name FROM products LIMIT 5` - **Root Cause**: Missing ordering instruction in context. - **Proposed Mutation**: Update the template for "Product Sales" to include ordering. ### Query 3: "Get users older than 30" - **Error Category**: `[GoldenDataError]` - **Expected SQL**: `SELECT * FROM users WHERE age >> 30` (Syntax error `>>` in golden SQL) - **Actual SQL**: `SELECT * FROM users WHERE age > 30` - **Root Cause**: Invalid syntax in golden dataset. - **Proposed Mutation**: None. Flag to user to fix the evaluation dataset. ``` 4. **Save Report**: You **MUST** physically write the report file to `autoctx/experiments//hillclimb/gap_analysis_vN.md`. If you are processing in batches, ensure you append to this file until all failed queries are documented. Do not merely output it in chat; it must exist on the file system. 5. **Log in State Tracking**: - Update `autoctx/state.md` to record the mapping for Loop `vN` (Base Context <-> Eval Report <-> Gap Analysis). 6. **Human-in-the-Loop Review**: - Inform the user that the Gap Analysis report has been successfully written to disk. - Ask the user if they want to review, make any corrections, or add manual feedback directly to the file before proceeding to Phase 2 (Context Mutation). - Wait for user confirmation before starting Phase 2. --- ### 3. Phase 2: Context Mutation Refer to [context-generation-guide](../context-generation-guide/SKILL.md) for how to edit a ContextSet. 1. **Validation**: Verify that `gap_analysis_vN.md` exists and contains findings. Verify the base ContextSet file exists. If missing, STOP and inform the user. 2. **Analyze Gap Report & Determine Fixing Strategy**: - Read `gap_analysis_vN.md` to identify what needs to be fixed. - **Fixing Strategy Guidelines**: - **Conciseness**: Try to use *less context* to cover *more scenarios*. Avoid adding redundant or hyper-specific templates for every single edge case. - **Generalizability**: Prefer solutions that generalize well (e.g., use a `facet` for a column definition rather than a specific `template` for every query using that column). - **Supported Types**: Support mutations for `template`, `facet`, and `value_search` types. 3. **Apply Mutations**: - **Copy the Base Context**: Copy the base ContextSet file to the new destination: `autoctx/experiments//hillclimb/improved_context_vN.json`. - **Generate New Items**: For any new context items identified in the fixing strategy (for "add" operations): - **Invoke the `context-generation-guide` skill** to produce the final parameterized items. - Provide the identified candidates to that skill. - That skill will handle phrase extraction, parameterization, and constructing the valid JSON structure. - **Validate New Items**: - **Templates**: Run generated SQL examples via `-execute-sql` (use dummy values for placeholders) to verify syntax. - **Others**: Cross-check table/column references against the schema via `-list-schemas`. - **Apply Mutations**: Call the `mutate_context_set` MCP tool passing the **new** file path as `file_path` and mutations as `mutations_json` to mutate the context set. 4. **Validate**: Call `validate_context_set` on `improved_context_vN.json`. If invalid, fix each issue via `mutate_context_set` and re-validate until clean. Stop after two failed attempts and surface remaining issues to the user. 5. **Log in State Tracking**: - Update `autoctx/state.md` to include the output path of `improved_context_vN.json` for Loop `vN`. --- ### 4. Validation & Upload Advice 1. **Summarize Improvements**: Tell the user what was changed (e.g., added 2 facets, updated 1 template). 2. **Upload Instructions**: - **Read Database Details**: Read `autoctx/tools.yaml` (or `db_config.yaml`) to fetch the specific project, location, and instance/cluster details for the active database. - **Generate URL**: Call the `generate_upload_url` tool passing the extracted values to provide the direct console link to the user. - Present the local file path to `improved_context_vN.json` and the generated console link together in a single clear message. 3. **Instruct Next Step Evaluation**: - Instruct the user to run evaluation using the evaluating workflow on this new ContextSet to see if metrics improve. This will start Loop `N+1`. --- ## Output Upon successful completion, the workspace must contain: - `autoctx/experiments//hillclimb/gap_analysis_vN.md` - `autoctx/experiments//hillclimb/improved_context_vN.json` - Updated `autoctx/state.md` summarizing the run loop. --- ## Logging State Example (`autoctx/state.md`) When updating `autoctx/state.md`, preserve the existing `## Active Database` section and append or update the `Hill-Climbing Run Log` section: ```markdown ## Active Experiment: my-exp-1 ## Hill-Climbing Run Log ### Loop: v1 - **Base Context**: `baseline_context.json` - **Eval Report Path**: `autoctx/experiments/my-exp-1/eval_reports//` (containing `configs.csv`, `evals.csv`, etc.) - **Gap Analysis**: `autoctx/experiments/my-exp-1/hillclimb/gap_analysis_v1.md` - **Mutated Context**: `autoctx/experiments/my-exp-1/hillclimb/improved_context_v1.json` ``` > [!IMPORTANT] > **Tool Modification Rule**: Always use the `mutate_context_set` tool for all ContextSet changes. Pass mutation payloads directly to the tool — it handles all file I/O internally. **Do not read the target context set file beforehand**.