Counterfactual Detection Eval

counterfactual-detection-eval

I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews — O'Neill et al. (2021) (arXiv:2104.06893, 2021)

What this evaluates

Evaluates a model's ability to detect counterfactual statements in product reviews. It probes robustness to selection bias from clue phrases, cross-lingual transfer via machine translation, and the effectiveness of different sentence encoders and classifiers on imbalanced binary classification tasks.

Datasets

• Multilingual Counterfactual Detection Dataset (Amazon Reviews) — total 24000; splits: train (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Standard binary classification metric.
• Matthew's Correlation Coefficient (MCC) — range: [-1, 1]
  • Correlation coefficient between observed and predicted binary classifications. Accounts for class imbalance and incorporates all correlations within the confusion matrix.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of total instances.
• Precision — range: [0, 1]
  • Proportion of true positive predictions among all positive predictions.
• Recall — range: [0, 1]
  • Proportion of true positive predictions among all actual positive instances.

Input / output format

Input: Single sentence text string.

Output: Binary label indicating whether the sentence expresses a counterfactual (positive) or not (negative).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum((t == 1 and p == 1) for t, p in zip(y_true, y_pred))
    fp = sum((t == 0 and p == 1) for t, p in zip(y_true, y_pred))
    fn = sum((t == 1 and p == 0) for t, p in zip(y_true, y_pred))
    tn = sum((t == 0 and p == 0) for t, p in zip(y_true, y_pred))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    acc = (tp + tn) / (tp + fp + fn + tn)
    denom = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn)) ** 0.5
    mcc = (tp * tn - fp * fn) / denom if denom > 0 else 0
    return {'F1': f1, 'MCC': mcc, 'Accuracy': acc, 'Precision': prec, 'Recall': rec}

Common pitfalls

• Accuracy can be highly misleading on this dataset due to severe class imbalance (underrepresented positive class).
• Models may memorize clue phrases used for data selection rather than learning semantic counterfactual cues, leading to overfitting when clues are masked or absent in test data.
• Japanese tokenization (e.g., MeCab) can split or remove verb/adjective inflections that contain counterfactual clues, artificially lowering performance for transformer models using BPE.

Evidence (verbatim from paper)

F1, Matthew's Correlation Coefficient (MCC; Boughorbel et al., 2017), and accuracy are used as evaluation metrics. MCC (∈ [-1,1]) accounts for class imbalance and incorporates all correlations within the confusion matrix (Chicco and Jurman, 2020). Accuracy may be misleading in highly imbalanced datasets because a simple classification of all instances to the majority class has a high accuracy. However, for consistency with prior work, we report all three evaluation metrics in this paper. All the reported results are averaged over at least 3 independently trained models initialised with the same hyperparameter values.

Citation

@misc{oneill2021counterfactual,
  title={I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews},
  author={O'Neill et al. (2021)},
  year={2021},
  note={arXiv:2104.06893}
}

• arXiv: 2104.06893