Commit Message Completion Eval

commit-message-completion-eval

From Commit Message Generation to History-Aware Commit Message Completion — Eliseeva et al. (2023) (arXiv:2308.07655, 2023)

What this evaluates

Evaluates how well models generate or complete commit messages given code diffs and optional historical context. It probes the model's ability to follow coding conventions, match ground truth exactly, and maintain semantic similarity under varying context lengths.

Datasets

• CMG_test — total ?; splits: test (-1); repo https://github.com/JetBrains-Research/commit_message_generation

Metrics

• B-Norm — range: [0, 1]
  • Normalized BLEU score variant measuring n-gram overlap between predicted and ground truth commit messages.
• Edit Similarity — range: [0, 1]
  • Ratio of matching characters or tokens after optimal alignment, measuring surface-level string similarity.
• ExactMatch@1 (primary) — range: [0, 1]
  • Fraction of instances where the top-1 generated commit message exactly matches the ground truth.
• ExactMatch@2 — range: [0, 1]
  • Fraction of instances where the ground truth appears in the top-2 generated candidates.

Input / output format

Input: Code diff (and optionally previous commit messages for history) plus a user-typed commit message prefix (for completion) or empty prefix (for generation).

Output: A single commit message string.

Scoring recipe

def compute_metrics(predictions, golds):
    em1 = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    em2 = sum(1 for p, g in zip(predictions, golds) if g in p or p in g) / len(golds)
    b_norm = compute_normalized_bleu(predictions, golds, n=4)
    ed_sim = compute_edit_similarity(predictions, golds)
    return {'B-Norm': b_norm, 'EdSim': ed_sim, 'EM@1': em1, 'EM@2': em2}

Common pitfalls

• Evaluating on filtered subsets (e.g., First Sentence, Verb-Direct Object) inflates metrics and fails to reflect real-world commit diversity.
• Confusing the generation setting (0% context) with the completion setting (25% context), as metrics trend differently and completion is inherently easier.

Evidence (verbatim from paper)

From Table III, we observe that B-Norm and Edit Similarity metrics across all the models and settings increase when adding history.

Citation

@misc{eliseeva2023commit,
  title={From Commit Message Generation to History-Aware Commit Message Completion},
  author={Eliseeva et al. (2023)},
  year={2023},
  note={arXiv:2308.07655}
}

• arXiv: 2308.07655