Rumoureval Eval

rumoureval-eval

SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours — Derczynski et al. (2017) (arXiv:1704.05972, 2017)

What this evaluates

Evaluates a model's ability to determine the veracity of social media rumours and classify the discourse stance of replies within a conversation tree. It probes contextual discourse analysis, stance detection, and truthfulness judgment in noisy, interactive text.

Datasets

• RumourEval (SemEval-2017 Task 8) — total ?; splits: train (-1), test (-1)

Metrics

• classification accuracy (primary) — range: [0, 1]
  • Ratio of correctly classified stance labels (support, deny, query, comment) to the total number of instances.
• macroaveraged accuracy (primary) — range: [0, 1]
  • Ratio of correctly predicted veracity labels (true/false) to total instances, averaged per class.
• combined veracity score — range: [0, 1]
  • Final score calculated by normalising the RMSE of confidence differences (rho) and multiplying by macroaveraged accuracy: (1 - rho) * accuracy.

Input / output format

Input: A rumour claim and a reply (or conversation thread) in a tree-structured social media discussion.

Output: For SDQC: one of four stance labels (support, deny, query, comment). For Veracity: one of two labels (true, false).

Scoring recipe

# SDQC Stance Classification
sdqc_correct = sum(1 for p, g in zip(sdqc_preds, sdqc_golds) if p == g)
sdqc_acc = sdqc_correct / len(sdqc_golds)

# Veracity Prediction
veracity_correct = sum(1 for p, g in zip(veracity_preds, veracity_golds) if p == g)
veracity_acc = veracity_correct / len(veracity_golds)

# Confidence RMSE & Combined Score
correct_preds = [(sys_c, ref_c) for p, g, sys_c, ref_c in zip(veracity_preds, veracity_golds, sys_confs, ref_confs) if p == g]
rmse_rho = sqrt(mean((s - r)**2 for s, r in correct_preds)) if correct_preds else 1.0
# Incorrect examples get RMSE of 1.0
final_score = (1 - rmse_rho) * veracity_acc

Common pitfalls

• The class distribution is heavily skewed towards 'comment', which can inflate naive accuracy if not handled.
• The final veracity score combines label accuracy and confidence RMSE, so optimizing only for label accuracy may yield a lower final score than expected.
• The paper's formula acc = (1 - \rho)acc appears to be a typo for the combined score; participants must normalize RMSE before applying it.

Evidence (verbatim from paper)

SDQC stance classification: The evaluation of the SDQC needed careful consideration, as the distribution of the categories is clearly skewed towards comments. Evaluation is through classification accuracy. Veracity prediction: The evaluation of the predicted veracity, which is either true or false for each instance, was done using macroaveraged accuracy, hence measuring the ratio of instances for which a correct prediction was made. Additionally, we calculated RMSE $\rho$ for the difference between system and reference confidence in correct examples and provided the mean of these scores. Incorrect examples have an RMSE of 1. This is normalised and combined with the macroaveraged accuracy to give a final score; e.g. $acc = (1 - \rho)acc$ .

Citation

@misc{derczynski2017rumoureval,
  title={SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours},
  author={Derczynski et al. (2017)},
  year={2017},
  note={arXiv:1704.05972}
}

• arXiv: 1704.05972