LLM Generation Eval

llm-generation-eval

Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference — Wang et al. (2023) (arXiv:2303.04673, 2023)

What this evaluates

Evaluates the utility of LLM text generation across code, math, and summarization tasks under inference budget constraints. It probes whether jointly tuning generation hyperparameters (e.g., temperature, top-p, number of responses) improves task performance compared to default or benchmark configurations.

Datasets

• APPS — total ?; splits: tune (20), test (-1)
• HumanEval — total ?; splits: tune (20), test (-1)
• MATH — total ?; splits: tune (20), test (-1)
• XSum — total ?; splits: tune (60), test (-1)

Metrics

• pass_rate (code) (primary) — range: [0, 1]
  • Binary score: 1 if any generated response passes the provided test cases, 0 otherwise.
• success (MATH) — range: [0, 1]
  • Binary score: 1 if any returned chain-of-thought response has an equivalent final answer to the ground truth, 0 otherwise.
• success Vote (MATH) — range: [0, 1]
  • Binary score: 1 if the response derived from majority voting has an equivalent final answer to the ground truth, 0 otherwise.
• Rouge-2 (XSum) — range: [0, 100]
  • ROUGE-2 F1 score computed on the top response after reranking all generated responses by their mean log probabilities.

Input / output format

Input: Text prompts containing coding problems, function definitions, math questions, or news articles, optionally with few-shot examples or specific template instructions.

Output: Generated text responses (Python code, step-by-step math derivations with final answer, or news summaries).

Scoring recipe

def score(predictions, gold, dataset, metric_type):
    if dataset in ['APPS', 'HumanEval']:
        return 1 if any(pass_test_cases(pred, gold['tests']) for pred in predictions) else 0
    if dataset == 'MATH':
        if metric_type == 'success':
            return 1 if any(extract_final_answer(pred) == gold['answer'] for pred in predictions) else 0
        if metric_type == 'success Vote':
            return 1 if majority_vote(predictions) == gold['answer'] else 0
    if dataset == 'XSum':
        reranked = sort_by_mean_logprob(predictions)
        return rouge2_score(reranked[0], gold['summary'])

Common pitfalls

• Reranking criterion (mean log probability) for XSum is not aligned with the final ROUGE-2 evaluation metric, potentially masking tuning benefits.
• Increasing inference budget per trial reduces the total number of optimization trials, which can cause performance drops if the optimization budget is fixed.
• Default model recommendations (e.g., code-davinci-002 for code) may be suboptimal after joint hyperparameter tuning.

Evidence (verbatim from paper)

For each code generation instance, as long as any response passes the test cases, the score that EcoOptiGen receives for this instance is 1 and 0 otherwise. For MATH, we consider two ways of evaluation. In Section 4.2 to Section 4.4, we define "success" as: if one of the returned chain-of-thought responses has an equivalent final answer with the ground truth, EcoOptiGen receives 1 for this instance and 0 otherwise. In Section 4.5, we define "success Vote" as: if the response based on majority voting has an equivalent final answer with the ground truth, EcoOptiGen receives 1 for this instance and 0 otherwise. For XSum, we use 'best_of' to rerank the generated responses by their mean log probabilities and use the Rouge-2 score for the top response (Lin, 2004).

Citation

@misc{wang2023costeffective,
  title={Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2303.04673}
}

• arXiv: 2303.04673