Civl Eval

civl-eval

Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances — Rishu Kumar Singh et al. (2025) (arXiv:2511.14693, 2025)

What this evaluates

This benchmark evaluates multimodal complaint analysis by measuring a model's ability to jointly process multi-turn textual dialogues and accompanying images to classify fine-grained aspects and severity levels of customer grievances. It probes cross-modal alignment, multi-label classification, and robustness to class imbalance and subjective tone variations.

Datasets

• CIViL — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/sarmistha-D/VALOR

Metrics

• Accuracy — range: [0, 1]
  • Proportion of correctly predicted labels out of total instances. Computed independently for Aspect Classification (ACD) and Severity Classification (SD) tasks.
• macro F1-score (primary) — range: [0, 1]
  • Unweighted mean of the F1-score for each class, treating all classes equally regardless of frequency. Computed independently for ACD and SD tasks.

Input / output format

Input: Multi-turn customer complaint dialogues containing both textual messages and associated images (visual evidence).

Output: Multi-label classification predictions for two dimensions: Aspect Classification (ACD) and Severity Classification (SD).

Scoring recipe

from sklearn.metrics import accuracy_score, f1_score

def compute_metrics(y_true, y_pred):
    # y_true and y_pred are binary multi-label matrices (n_samples, n_classes)
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    return {'accuracy': acc, 'macro_f1': f1}

Common pitfalls

• Class imbalance: Over-representation of dominant aspects (e.g., 'software') and underrepresentation of others (e.g., 'price') can skew predictions and reduce generalization to low-frequency categories.
• Subjective severity interpretation: Variability in user tone or emotionally neutral expressions can lead models to underestimate or misclassify severity levels.
• Multi-aspect recognition: Complex complaints involving multiple issues require the model to correctly map distinct aspect-severity pairs rather than predicting a single dominant label.

Evidence (verbatim from paper)

Evaluation is performed using Accuracy and macro F1-score, computed independently for both ACD and SD tasks to provide a comprehensive assessment of model performance across the two fine-grained complaint dimensions.

Citation

@misc{singh2025valor,
  title={Talk, Snap, Complain: Validation-Aware Multimodal Expert Framework for Fine-Grained Customer Grievances},
  author={Rishu Kumar Singh et al. (2025)},
  year={2025},
  note={arXiv:2511.14693}
}

• arXiv: 2511.14693