Esperanto Eval

esperanto-eval

ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination — Ayoobi et al. (2024) (arXiv:2409.14285, 2024)

What this evaluates

Evaluates the robustness of AI-generated text detectors against back-translation manipulations. It probes whether detectors can maintain high true positive rates when AI-generated text is translated to intermediate languages and back-translated to English, preserving semantics while evading detection.

Datasets

• ESPERANTO — total 792000; splits: full (-1); repo https://github.com/navid-aub/Esperanto-Dataset

Metrics

• True Positive Rate (TPR) (primary) — range: [0, 1]
  • TPR = TP / (TP + FN), where TP is the number of AI-generated texts correctly identified as AI, and FN is the number of AI-generated texts incorrectly classified as human. Measures the detector's ability to catch AI text after manipulation.

Input / output format

Input: English text instances (news articles, paper abstracts, Reddit QA responses, product reviews), typically truncated to ~300 words at sentence boundaries. Some instances are back-translated via intermediate languages.

Output: Binary classification label (AI-generated vs. Human-written) or detection probability score from the evaluated detector.

Scoring recipe

def compute_tpr(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 1 and g == 1)
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p == 0 and g == 1)
    return tp / (tp + fn) if (tp + fn) > 0 else 0.0

Common pitfalls

• Truncating text mid-sentence artificially lowers AI detectability; the dataset enforces sentence-boundary truncation to maintain fairness.
• Back-translation can alter semantics if not carefully combined; the authors use word error rate (WER) to merge back-translated outputs and preserve original meaning.
• Detectors may exhibit domain bias; the dataset spans four distinct writing styles (news, scientific, informal QA, product reviews) to mitigate this.

Evidence (verbatim from paper)

The paper introduces back-translation as a novel evasion technique to manipulate AI-generated text, reducing the true positive rate (TPR) of existing AI detection systems by up to 52%—with one detector (RADAR) showing a 52% drop in TPR and a minimal 1.85% TPR decline in a robust countermeasure.

Citation

@misc{ayoobi2024esperanto,
  title={ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination},
  author={Ayoobi et al. (2024)},
  year={2024},
  note={arXiv:2409.14285}
}

• arXiv: 2409.14285