Sged Eval

sged-eval

SGED: A Benchmark dataset for Performance Evaluation of Spiking Gesture Emotion Recognition — Wang et al. (2023) (arXiv:2304.14714, 2023)

What this evaluates

Evaluates a model's ability to recognize human emotions (neutral, negative, positive) from dynamic gesture videos. It specifically probes robustness to class imbalance and performance under low-light/high-motion conditions using multimodal inputs.

Datasets

• SGED — total ?; splits: test (-1); repo https://github.com/201528014227051/SGED

Metrics

• Accuracy (primary) — range: [0, 1]
  • The proportion of correctly classified samples out of the total number of samples.
• Weighted Precision — range: [0, 1]
  • Precision averaged across classes, weighted by the number of true instances for each class.
• Weighted Recall — range: [0, 1]
  • Recall averaged across classes, weighted by the number of true instances for each class.
• Weighted F1 — range: [0, 1]
  • Harmonic mean of weighted precision and weighted recall, averaged across classes weighted by support.

Input / output format

Input: Multimodal gesture data consisting of RGB video frames and high-temporal-resolution event stream data (from DAVIS346 sensor) representing dynamic hand/body gestures.

Output: Classification into one of three emotion categories: neutral, negative, or positive.

Scoring recipe

def compute_metrics(y_true, y_pred, classes):
    acc = np.mean(y_true == y_pred)
    w_prec, w_rec, w_f1 = [], [], []
    for c in classes:
        tp = np.sum((y_pred == c) & (y_true == c))
        fp = np.sum((y_pred == c) & (y_true != c))
        fn = np.sum((y_pred != c) & (y_true == c))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        w_prec.append(prec * np.sum(y_true == c))
        w_rec.append(rec * np.sum(y_true == c))
        w_f1.append(f1 * np.sum(y_true == c))
    total = np.sum([np.sum(y_true == c) for c in classes])
    return acc, sum(w_prec)/total, sum(w_rec)/total, sum(w_f1)/total

Common pitfalls

• Ignoring class imbalance: The dataset has significantly more positive samples than neutral/negative. Evaluating solely on standard accuracy can be misleading.
• Using unweighted metrics: The authors explicitly recommend weighted precision, recall, and F1 to account for the uneven distribution of emotion categories.
• Assuming fixed splits: The paper notes multiple splitting methods were provided, so results may vary depending on the chosen data partition.

Evidence (verbatim from paper)

Because gesture emotion recognition can be seen as a classification task, the accuracy and confusion matrix are used to evaluate the model. In order to take into account sample imbalance in the evaluation of model performance, we also used weighted precision, weighted recall, and weighted F1 score as the evaluation criterion.

Citation

@misc{wang2023sged,
  title={SGED: A Benchmark dataset for Performance Evaluation of Spiking Gesture Emotion Recognition},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2304.14714}
}

• arXiv: 2304.14714