Gpo Prompt Optimization Eval

gpo-prompt-optimization-eval

Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers — Tang et al. (2024) (arXiv:2402.17564, 2024)

What this evaluates

Evaluates the effectiveness of LLM-based prompt optimizers across complex reasoning, knowledge-intensive, and common NLP tasks. It measures how much optimized prompts improve model performance compared to baseline prompts and other optimization methods.

Datasets

• Big-Bench Hard (BBH) — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MMLU — total ?; splits: test (-1)
• WSC — total ?; splits: test (-1)
• WebNLG — total ?; splits: test (-1)

Metrics

• average accuracy (primary) — range: [0, 100] percent
  • Calculated as the number of correct predictions divided by the total number of instances, multiplied by 100. Reported as the average across all subtasks for BBH and MMLU, and directly for GSM8K.
• ROUGE-L — range: [0, 1]
  • Computes the longest common subsequence between the reference and generated text to measure overlap. Used for WSC and WebNLG following Wang et al. (2022).

Input / output format

Input: Task prompt (generated by the optimizer or baseline) concatenated with the input question or example from the dataset.

Output: Model-generated text or discrete answer corresponding to the dataset's expected output format.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(golds)) * 100

def compute_rouge_l(predictions, references):
    scores = [rouge_l_score(p, r) for p, r in zip(predictions, references)]
    return sum(scores) / len(scores)

Common pitfalls

• Experiments use sampled subsets of each dataset due to computational limitations, not the full benchmarks.
• Results are averaged over three independent runs, but standard deviations or confidence intervals are not reported in the main tables.
• ROUGE-L is used for WSC and WebNLG, which may not align perfectly with exact-match accuracy used for reasoning tasks.

Evidence (verbatim from paper)

Evaluation Metrics. We report the average accuracy of all the subtasks for BBH and MMLU following Suzgun et al. (2023) and Hendrycks et al. (2021), accuracy for GSM8K following Cobbe et al. (2021), ROUGE-LLin (2004) for WSC and WebNLG following Wang et al. (2022).

Citation

@misc{tang2024gpo,
  title={Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2402.17564}
}

• arXiv: 2402.17564