qhjqhj00 multilogbench-eval
Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs — Zhong et al. (2026) (arXiv:2604.17529, 2026)
What this evaluates
Evaluates LLMs' ability to generate appropriate logging statements for code callables across six programming languages, testing both snapshot-based code understanding and revision-history-based code evolution contexts.
Datasets
- MultiLogBench — total ?; splits: repository-snapshot (-1), revision-history (-1), transformed (-1); repo https://github.com/logresearch/MultiLogBench
Metrics
exact-match accuracy(primary) — range: [0, 1]- Percentage of generated logging statements that exactly match the developer-authored gold statement.
Input / output format
Input: A callable (function/method) extracted from a frozen repository snapshot or a parent revision, with the target logging statement removed and formatting gaps repaired.
Output: A single logging statement string to be inserted at the target site.
Scoring recipe
def score(predictions, golds):
correct = 0
for pred, gold in zip(predictions, golds):
if pred.strip() == gold.strip():
correct += 1
return correct / len(golds)
Common pitfalls
- Confusing repository-snapshot data (pre-existing logs) with revision-history data (log-introducing edits)
- Failing to account for framework-specific logging APIs vs. standard ones
- Overestimating performance due to pretraining contamination on GitHub-sourced code
Evidence (verbatim from paper)
Repository-snapshot data serves as the primary evaluation source. Its instances are mined from a frozen snapshot of each selected repository, and each target logging statement is a developer-authored statement that already exists in that revision.
Citation
@misc{zhong2026multilogbench,
title={Single-Language Evidence Is Insufficient for Automated Logging: A Multilingual Benchmark and Empirical Study with LLMs},
author={Zhong et al. (2026)},
year={2026},
note={arXiv:2604.17529}
}
- arXiv: 2604.17529