Openturingbench Eval

openturingbench-eval

OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution — La Cava et al. (2025) (arXiv:2504.11369, 2025)

What this evaluates

Evaluates the capability of models to detect machine-generated text and attribute it to specific authors or models across diverse scenarios, including mixed human-machine text, out-of-domain content, and outputs from unseen LLMs.

Datasets

• OpenTuringBench — total ?; splits: train (-1), test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Computed from the confusion matrix where 'MACHINE' is the positive class. Uses a weighted-average approach across classes/tasks to account for class size variations, aligning with MGTBench defaults.

Input / output format

Input: Text snippets (human-written, machine-generated, or mixed) with associated metadata for authorship/model attribution tasks.

Output: Binary classification label ('HUMAN' or 'MACHINE') for detection, or author/model identifier for attribution.

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 'MACHINE' and g == 'MACHINE')
    fp = sum(1 for p, g in zip(predictions, gold) if p == 'MACHINE' and g != 'MACHINE')
    fn = sum(1 for p, g in zip(predictions, gold) if p != 'MACHINE' and g == 'MACHINE')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return f1

Common pitfalls

• Detectors must be fine-tuned for 10 epochs on the OpenTuringBench train set before evaluation to ensure fair comparison, especially for authorship attribution.
• The 'MACHINE' class is strictly treated as the positive class for all metric calculations, which reverses standard human-positive conventions.
• Evaluation uses a weighted-average approach across classes/tasks rather than macro-averaging to handle inherent class imbalances.

Evidence (verbatim from paper)

We assessed the performance of OTBDetector and competing methods through standard metrics derived from the confusion matrices obtained in the various TT and AA tasks (cf. Sect. 2.3). These include precision (P), recall (R), and F1-score (F1), where outcomes 'MACHINE' are regarded as instances of the positive class. The scores were computed using a weighted-average approach, which is commonly used to account for variations in class sizes and to align with the default settings in MGTBench.

Citation

@misc{lacava2025openturingbench,
  title={OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution},
  author={La Cava et al. (2025)},
  year={2025},
  note={arXiv:2504.11369}
}

• arXiv: 2504.11369