Translation Proofreading Eval

translation-proofreading-eval

Design of intelligent proofreading system for English translation based on CNN and BERT — Liu et al. (2025) (arXiv:2506.04811, 2025)

What this evaluates

Evaluates a hybrid CNN-BERT model's ability to detect and correct errors in English translations. It measures performance across different architectural hyperparameters (kernel size, batch normalization) and linguistic granularity levels.

Datasets

• WMT English-German Parallel Corpus Dataset — total ?; splits: (unstated)
• Open Subtitles Dataset — total ?; splits: (unstated)

Metrics

• Precision (%) — range: percent
  • Ratio of correctly identified positive cases to all cases predicted as positive. Reported as a percentage.
• Recall (%) — range: percent
  • Ratio of correctly identified positive cases to all actual positive cases. Reported as a percentage.
• F1-Score (%) (primary) — range: percent
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Reported as a percentage.
• Accuracy (%) — range: percent
  • Ratio of all correct predictions to total predictions. Reported as a percentage.
• RMSE (%) — range: percent
  • Root Mean Square Error: sqrt(mean((y_true - y_pred)^2)). Reported as a percentage.
• MSE (%) — range: percent
  • Mean Squared Error: mean((y_true - y_pred)^2). Reported as a percentage.
• MAE (%) — range: percent
  • Mean Absolute Error: mean(|y_true - y_pred|). Reported as a percentage.

Input / output format

Input: Translated text or sentence pairs (source-target) for error detection and correction.

Output: Corrected translation or error labels/masks.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = (tp + tn) / len(y_true)
    mse = sum((t - p)**2 for t, p in zip(y_true, y_pred)) / len(y_true)
    rmse = mse**0.5
    mae = sum(abs(t - p) for t, p in zip(y_true, y_pred)) / len(y_true)
    return precision*100, recall*100, f1*100, accuracy*100, rmse*100, mse*100, mae*100

Common pitfalls

• Mixes classification metrics (Precision/Recall/F1/Accuracy) with regression-style error metrics (RMSE/MSE/MAE) for the same proofreading task without clarifying the target variable or thresholding strategy.
• High accuracy scores (e.g., 90%) are reported alongside low F1-scores (~50%), indicating potential severe class imbalance or thresholding issues not addressed in the evaluation protocol.
• Metrics are reported at multiple linguistic granularity levels (Phrase to Chapter) but the aggregation method across these levels is not specified.

Evidence (verbatim from paper)

The network attains a 75.32% precision that represent accuracy for the identification of meaningful data points by the usage of kernel size 1. The recall is 82.36% which is good at recognizing all relevant circumstances. The F1-Score stands at 85.31% as well as the overall accuracy is 80.54%.

Citation

@misc{liu2025design,
  title={Design of intelligent proofreading system for English translation based on CNN and BERT},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.04811}
}

• arXiv: 2506.04811