# 2600 Data Driven Optimize Output Analysis E58bea19 > Guide to Analyzing Data-Driven Optimize Output - Skill: `tools-only/2600-data-driven-optimize-output-analysis-e58bea19` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add tools-only/2600-data-driven-optimize-output-analysis-e58bea19` - Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/2600-data-driven-optimize-output-analysis-e58bea19/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: tools-only (https://skillmd.com/u/tools-only) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/tools-only/2600-data-driven-optimize-output-analysis-e58bea19 --- # Guide to Analyzing Data-Driven Optimize Output ## 1. Prerequisites: GCS Authentication Before using the Python tools described in this guide, you must authenticate your local environment with Google Cloud. The tools need to access the output files stored in a GCS bucket, which requires proper permissions. ## 2. Introduction After a Data-Driven Optimize (Prompt Optimizer) job completes, it writes a number of files to the specified output path. This can be a Google Cloud Storage (GCS) bucket or a local directory. This document explains what these files are and how to use the provided tools to analyze them. ## 3. Data-Driven Optimize Output Files The output directory contains several JSON files that capture the results of the optimization. - **`optimized_results.json`**: Contains the single best prompt template found. The entries in the following pairs of files have corresponding orders, and their contents can be zipped together for detailed analysis: - **Training Results During Optimization Process**: - **`templates.json`**: Contains all templates evaluated during the optimization process, along with their summary metrics. - **`eval_results.json`**: Contains detailed, per-row training results on minibatches during the optimization process. This file is for manual, in-depth analysis and is not read by the analysis tools. - **Initial and Best Templates on Full Test Set**: - **`test_templates.json`**: Contains your original prompt and the best prompt, along with their summary metrics on the full test dataset. The best prompt is selected based on the metric from training only. - **`test_eval_results.json`**: Contains the detailed, per-row evaluation scores for the original and best prompts on the full test dataset. This file is for manual, in-depth analysis and is not read by the analysis tools. - **Eval of All Candidates on Full Test Set**: - **`all_candidates_full_test_templates.json`**: A list of *all* prompt candidates, along with their summary metrics on the full test set. - **`all_candidates_full_test_eval_results.json`**: Contains detailed, per-row evaluation results for each candidate prompt on the full test set. This file is for manual, in-depth analysis and is not read by the analysis tools. - **Eval of All Candidates on Full Training Set**: - **`all_candidates_full_train_templates.json`**: A list of *all* prompt candidates, along with their summary metrics on the full training set. - **`all_candidates_full_train_eval_results.json`**: Contains detailed, per-row evaluation results for each candidate prompt on the full training set. This file is for manual, in-depth analysis and is not read by the analysis tools. ### Controlling Output File Generation Not all output files are generated by default. Their creation depends on your job's configuration. This is important because the analysis tools are designed to handle missing files gracefully, but you may want to enable certain outputs for a more complete analysis. Here are the key flags that control which files are generated: - **Providing Test Data (`test_input_data_path`)**: This parameter is **optional but highly recommended**. When you provide a test dataset, the `test_templates.json` and `test_eval_results.json` files are generated, which provide a final comparison between your original prompt and the best-optimized prompt. Without a test set, you cannot accurately measure the generalization performance of your prompts. - **`eval_all_candidates_on_full_test`**: This is an **optional** boolean flag. When set to `True` (and a test set is provided), the job will generate `all_candidates_full_test_templates.json` and `all_candidates_full_test_eval_results.json`. These files allow for a deep-dive analysis into the performance of *every* candidate on the test set. **Warning:** Enabling this flag can be computationally expensive and significantly increase the job's runtime. - **`eval_all_candidates_on_full_train`**: This is an **optional** boolean flag. When set to `True`, the job will generate `all_candidates_full_train_templates.json` and `all_candidates_full_train_eval_results.json`. These enable a detailed comparison of performance on the full training set versus the test set, and against batch training results, which is crucial for identifying overfitting. **Warning:** Enabling this flag can be computationally expensive and significantly increase the job's runtime. ## 4. Data File Structure Details **Key Fields:** - In `_templates.json` files (e.g., `all_candidates_full_train_templates.json`, `test_templates.json`): - **`step`**: The optimization step where this candidate was generated or selected. - **`metrics`**: This object contains aggregated evaluation results (e.g., `ROUGE_L/mean` or `exact_match/mean`). The specific metric name will vary depending on the `eval_metric` configured. It also contains the "uid" which is an unique identifier consistent across all result files. - **`prompt`**: The prompt string itself. - In `_eval_results.json` files (e.g., `all_candidates_full_train_eval_results.json`, `test_eval_results.json`): - **`summary_results`**: This object contains the aggregated metrics for the entire evaluation. - **`metrics_table`**: This object contains the per-row evaluation results, useful for detailed debugging. ## 5. Generating an Analysis Report The recommended way to analyze your results is to use a multi-step process involving the tools provided and your reasoning capabilities: 1. **Analyze Results:** First, call `analyze_data_driven_optimize_results(output_path, analysis_data_path="path/to/analysis_data.json")` to process the raw output files. Providing `analysis_data_path` is highly recommended as it saves the results into three separate files to avoid system size limits: - `analysis_data.json`: Core metadata (config, comparison, best prompt). - `analysis_data_metrics.json`: Detailed metrics for all candidates (no prompt text). - `analysis_data_prompts.json`: Mapping of top candidates' keys to full prompt texts. This tool returns a **minimal summary object** with the paths to the saved files and the best prompt score. 2. **Formulate Suggestions:** You should then analyze the `analysis_data` summary. Based on the guidelines in the **Data-Driven Optimize Tuning Guide**, formulate suggestions for the next run. This should result in a `suggested_config_data` dictionary, containing: - `"suggested_config"`: A dictionary of configuration modifications. - `"rationale"`: A string explaining the reasoning behind the suggestions. 3. **Generate HTML Report:** Finally, call `generate_html_report()` passing either the `analysis_data` object or the `analysis_data_path` obtained in Step 1, along with the `suggested_config_data`. Passing `analysis_data_path` is the most efficient way to handle large analysis results as it automatically re-joins the metadata, metrics, and prompts from the three split files. You can then open the generated HTML file in any web browser to view the full analysis. For a deeper understanding of how to interpret these results and iterate, please refer to the Data-Driven Optimize Tuning Guide.