2600 Data Driven Optimize Output Analysis E58bea19

Guide to Analyzing Data-Driven Optimize Output

1. Prerequisites: GCS Authentication

Before using the Python tools described in this guide, you must authenticate your local environment with Google Cloud. The tools need to access the output files stored in a GCS bucket, which requires proper permissions.

2. Introduction

After a Data-Driven Optimize (Prompt Optimizer) job completes, it writes a number of files to the specified output path. This can be a Google Cloud Storage (GCS) bucket or a local directory. This document explains what these files are and how to use the provided tools to analyze them.

3. Data-Driven Optimize Output Files

The output directory contains several JSON files that capture the results of the optimization.

• optimized_results.json: Contains the single best prompt template found.

The entries in the following pairs of files have corresponding orders, and their contents can be zipped together for detailed analysis:

• Training Results During Optimization Process:
  • templates.json: Contains all templates evaluated during the optimization process, along with their summary metrics.
  • eval_results.json: Contains detailed, per-row training results on minibatches during the optimization process. This file is for manual, in-depth analysis and is not read by the analysis tools.
• Initial and Best Templates on Full Test Set:
  • test_templates.json: Contains your original prompt and the best prompt, along with their summary metrics on the full test dataset. The best prompt is selected based on the metric from training only.
  • test_eval_results.json: Contains the detailed, per-row evaluation scores for the original and best prompts on the full test dataset. This file is for manual, in-depth analysis and is not read by the analysis tools.
• Eval of All Candidates on Full Test Set:
  • all_candidates_full_test_templates.json: A list of all prompt candidates, along with their summary metrics on the full test set.
  • all_candidates_full_test_eval_results.json: Contains detailed, per-row evaluation results for each candidate prompt on the full test set. This file is for manual, in-depth analysis and is not read by the analysis tools.
• Eval of All Candidates on Full Training Set:
  • all_candidates_full_train_templates.json: A list of all prompt candidates, along with their summary metrics on the full training set.
  • all_candidates_full_train_eval_results.json: Contains detailed, per-row evaluation results for each candidate prompt on the full training set. This file is for manual, in-depth analysis and is not read by the analysis tools.

Controlling Output File Generation

Not all output files are generated by default. Their creation depends on your job's configuration. This is important because the analysis tools are designed to handle missing files gracefully, but you may want to enable certain outputs for a more complete analysis.

Here are the key flags that control which files are generated:

• Providing Test Data (test_input_data_path): This parameter is optional but highly recommended. When you provide a test dataset, the test_templates.json and test_eval_results.json files are generated, which provide a final comparison between your original prompt and the best-optimized prompt. Without a test set, you cannot accurately measure the generalization performance of your prompts.

• eval_all_candidates_on_full_test: This is an optional boolean flag. When set to True (and a test set is provided), the job will generate all_candidates_full_test_templates.json and all_candidates_full_test_eval_results.json. These files allow for a deep-dive analysis into the performance of every candidate on the test set. Warning: Enabling this flag can be computationally expensive and significantly increase the job's runtime.

• eval_all_candidates_on_full_train: This is an optional boolean flag. When set to True, the job will generate all_candidates_full_train_templates.json and all_candidates_full_train_eval_results.json. These enable a detailed comparison of performance on the full training set versus the test set, and against batch training results, which is crucial for identifying overfitting. Warning: Enabling this flag can be computationally expensive and significantly increase the job's runtime.

4. Data File Structure Details

Key Fields:

• In _templates.json files (e.g., all_candidates_full_train_templates.json, test_templates.json):

  • step: The optimization step where this candidate was generated or selected.
  • metrics: This object contains aggregated evaluation results (e.g., ROUGE_L/mean or exact_match/mean). The specific metric name will vary depending on the eval_metric configured. It also contains the "uid" which is an unique identifier consistent across all result files.
  • prompt: The prompt string itself.

• In _eval_results.json files (e.g., all_candidates_full_train_eval_results.json, test_eval_results.json):

  • summary_results: This object contains the aggregated metrics for the entire evaluation.
  • metrics_table: This object contains the per-row evaluation results, useful for detailed debugging.

5. Generating an Analysis Report

The recommended way to analyze your results is to use a multi-step process involving the tools provided and your reasoning capabilities:

1. Analyze Results: First, call analyze_data_driven_optimize_results(output_path, analysis_data_path="path/to/analysis_data.json") to process the raw output files. Providing analysis_data_path is highly recommended as it saves the results into three separate files to avoid system size limits:
   • analysis_data.json: Core metadata (config, comparison, best prompt).
   • analysis_data_metrics.json: Detailed metrics for all candidates (no prompt text).
   • analysis_data_prompts.json: Mapping of top candidates' keys to full prompt texts. This tool returns a minimal summary object with the paths to the saved files and the best prompt score.
2. Formulate Suggestions: You should then analyze the analysis_data summary. Based on the guidelines in the Data-Driven Optimize Tuning Guide, formulate suggestions for the next run. This should result in a suggested_config_data dictionary, containing:
   • "suggested_config": A dictionary of configuration modifications.
   • "rationale": A string explaining the reasoning behind the suggestions.
3. Generate HTML Report: Finally, call generate_html_report() passing either the analysis_data object or the analysis_data_path obtained in Step 1, along with the suggested_config_data. Passing analysis_data_path is the most efficient way to handle large analysis results as it automatically re-joins the metadata, metrics, and prompts from the three split files.

You can then open the generated HTML file in any web browser to view the full analysis. For a deeper understanding of how to interpret these results and iterate, please refer to the Data-Driven Optimize Tuning Guide.