Mdpe Eval

mdpe-eval

MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics — Cai et al. (2024) (arXiv:2407.12274, 2024)

What this evaluates

Evaluates multimodal deception detection across video, audio, and text modalities, while also probing how individual differences—specifically personality traits and emotional expressivity—influence deceptive behavior and detection accuracy.

Datasets

• MDPE — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/cai-cong/MDPE

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly classified samples out of the total number of samples.
• Mean accuracy (A) — range: [0, 1]
  • A = 1 - (1/N^t) * Σ|Y_i^P - P_i|, where Y is predicted and P is ground truth. Functions as 1 minus MAE rather than standard classification accuracy.
• RMSE — range: other
  • Root Mean Square Error between predicted and ground truth emotion scores.

Input / output format

Input: Multimodal feature vectors per sample: acoustic, textual, and visual features. Optionally concatenated with personality scale scores and emotion expression feature vectors.

Output: Deception detection: softmax probabilities over 2 classes (truthful/deceptive). Personality/Emotion: continuous score vectors.

Scoring recipe

# Deception Accuracy
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
accuracy = correct / len(gold)

# Personality Mean Accuracy (A)
errors = sum(abs(y_pred - y_true) for y_pred, y_true in zip(preds, gold))
mean_acc = 1 - (errors / len(gold))

# Emotion RMSE
rmse = sqrt(mean((p - g)**2 for p, g in zip(preds, gold)))

Common pitfalls

• The 'Mean accuracy' metric for personality recognition is actually 1 minus MAE, not standard classification accuracy.
• Deception detection uses a per-sample answer split (5 for validation, 19 for training) rather than a standard subject-level train/val/test split.
• Emotional expression features are derived from an upstream emotion recognition model, so benchmark results are contingent on that model's quality.

Evidence (verbatim from paper)

For deception detection tasks, each sample comprises 24 answers. We randomly select 5 answers per sample (3 truthful, 2 deceptive) for validation, reserving the remaining 19 for training. All experiments are repeated five times with randomized initializations, and results report average performance to ensure statistical reliability. For personality and emotion recognition tasks, the dataset is split into 133 training samples, 40 validation samples, and 40 test samples; these tasks utilize root mean square error (RMSE) as the loss function. For deception detection, accuracy was selected as the evaluation metric. For personality recognition, we employed the mean accuracy (A), defined as follows: A = 1 - 1/N^t sum |Y_i^P - P_i|. This metric is widely adopted in personality recognition tasks. For emotion recognition, the root mean square error (RMSE) was used.

Citation

@misc{cai2024mdpe,
  title={MDPE: A Multimodal Deception Dataset with Personality and Emotional Characteristics},
  author={Cai et al. (2024)},
  year={2024},
  note={arXiv:2407.12274}
}

• arXiv: 2407.12274