Oms Keyword Gen Eval

oms-keyword-gen-eval

OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent — Bowen Chen et al. (2025) (arXiv:2507.02353, 2025)

What this evaluates

Evaluates an LLM agent's ability to generate high-quality, multi-objective ad keywords from product descriptions. It probes lexical and semantic alignment with product info, real-world campaign performance (clicks, conversions, cost), and the model's capacity for self-reflective refinement over multiple generation rounds.

Datasets

• OKG Benchmark Dataset — total 10000; splits: test (10000)

Metrics

• ROUGE-1 (primary) — range: [0, 1]
  • Word-level overlap between generated keywords and the product information. Higher values indicate better lexical coverage.
• BERTScore — range: [0, 1]
  • Semantic and lexical similarity between generated keywords and product information, computed using BERT embeddings.
• Clicks — range: other
  • Total number of campaign clicks for the generated keywords.
• Search Volume — range: other
  • Number of user searches for the generated keywords.
• Cost per Click (CpC) — range: other
  • Keyword efficiency metric indicating average cost per click. In figures, presented reversely (higher value = better performance).
• Competitor Score — range: other
  • Competitiveness of keywords as judged by the SSA platform.
• Conversion — range: other
  • Number of conversions achieved during the online A/B test.
• Click-Through Rate (CTR) — range: percent
  • Ratio of clicks to impressions during the online A/B test.
• Cost Per Acquisition (CPA) — range: other
  • Total cost divided by number of conversions during the online A/B test. Lower is better.

Input / output format

Input: Product information / description provided to the model.

Output: List of generated keywords.

Scoring recipe

def evaluate(predictions, gold):
    rouge1 = compute_rouge1(predictions, gold['product_info'])
    bertscore = compute_bertscore(predictions, gold['product_info'])
    clicks = gold['clicks']
    search_vol = gold['search_volume']
    cpc = gold['cpc']
    comp_score = gold['competitor_score']
    conversion = gold['conversions']
    ctr = gold['clicks'] / gold['impressions']
    cpa = gold['cost'] / gold['conversions']
    return {
        'ROUGE-1': rouge1, 'BERTScore': bertscore,
        'Clicks': clicks, 'Search Volume': search_vol,
        'CpC': cpc, 'Comp Score': comp_score,
        'Conversion': conversion, 'CTR': ctr, 'CPA': cpa
    }

Common pitfalls

• CpC is presented reversely in figures (higher value indicates better performance).
• Offline benchmark metrics in Table 3 are normalized, not raw values.
• Human preference scores use a 1–5 Likert scale, not accuracy or F1.

Evidence (verbatim from paper)

Metrics. We use following metrics: ROUGE-1 Lin ([2004]) compares the words overlap between generated keywords with the product information to compare the lexical coverage. BERTScore Zhang et al. ([2020]) considers both semantic and lexical similarity between generated keywords and the provided product information. Click, Search Volume, Cost per Click (CpC), and Competitor Score are four keyword performance metrics: Click and Search Volume represent the number of campaign clicks and user searches for keywords, CpC indicates keyword efficiency, and Competitor Score (judged by the SSA platform) reflects keywords’ competitiveness.

Citation

@misc{chen2025oms,
  title={OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent},
  author={Bowen Chen et al. (2025)},
  year={2025},
  note={arXiv:2507.02353}
}

• arXiv: 2507.02353