Dimabsa Eval

dimabsa-eval

DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis — Lee et al. (2026) (arXiv:2601.23022, 2026)

What this evaluates

Evaluates multilingual and multidomain aspect-based sentiment analysis by predicting continuous valence-arousal (VA) scores alongside aspect, opinion, and category extraction. It probes a model's ability to perform fine-grained dimensional sentiment regression and structured information extraction across diverse languages and domains.

Datasets

• DimABSA — total ?; splits: train (-1), test (-1)

Metrics

• RMSE_VA (primary) — range: [0, ∞)
  • Root Mean Square Error over the joint valence (V) and arousal (A) dimensions: sqrt(1/N * sum((V_p - V_g)^2 + (A_p - A_g)^2)).
• cF1 (primary) — range: [0, 1]
  • Continuous F1 score combining categorical exact match and continuous VA prediction error. A prediction is a continuous true positive (cTP) only if categorical elements exactly match gold, then cTP = 1 - dist(VA_p, VA_g), where dist is normalized Euclidean distance. cF1 is the harmonic mean of cPrecision and cRecall.

Input / output format

Input: Sentence text, optionally with aspect terms, opinion terms, or aspect categories depending on the subtask (DimASR, DimASTE, or DimASQP).

Output: Predicted continuous valence and arousal scores (V, A), and for DimASTE/DimASQP, extracted aspect, opinion, and category strings.

Scoring recipe

def score_rmse(preds, golds):
    errors = [(p['V']-g['V'])**2 + (p['A']-g['A'])**2 for p, g in zip(preds, golds)]
    return math.sqrt(sum(errors) / len(errors))

def score_cf1(preds, golds):
    cTP = 0
    for p, g in zip(preds, golds):
        if p['cat'] == g['cat']:
            dist = math.sqrt((p['V']-g['V'])**2 + (p['A']-g['A'])**2) / math.sqrt(128)
            cTP += 1 - dist
    cPrec = cTP / len(preds)
    cRec = cTP / len(golds)
    return 2 * cPrec * cRec / (cPrec + cRec) if (cPrec + cRec) > 0 else 0

Common pitfalls

• cF1 strictly requires exact categorical match before applying the continuous penalty; predictions with mismatched categories score 0 regardless of VA accuracy.
• RMSE_VA computes error jointly over both Valence and Arousal dimensions, not separately.
• Few-shot performance typically saturates around 32 examples; adding more shots does not guarantee further improvement.

Evidence (verbatim from paper)

Subtask 1. DimASR is evaluated by measuring the prediction error in the VA space using RMSE... Subtasks 2 & 3. DimASTE and DimASQP are evaluated using the continuous F1 (cF1) score, which unifies categorical and continuous evaluation.

Citation

@misc{lee2026dimabsa,
  title={DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2601.23022}
}

• arXiv: 2601.23022