Absa Sentiment Eval

absa-sentiment-eval

TravelBench : Exploring LLM Performance in Low-Resource Domains — Billa et al. (2025) (arXiv:2510.02719, 2025)

What this evaluates

Probes an LLM's ability to identify granular sentiment toward specific topics within a text, including custom labels like 'not mentioned' and 'wished for'. It tests fine-grained aspect-level classification rather than overall review sentiment.

Datasets

• TravelBench ABSA — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Computes the F1-score between prediction and ground truth labels using the Van Rijsbergen formulation.

Input / output format

Input: A hotel review text and a pre-defined set of topics (e.g., WiFi, pool, parking).

Output: Sentiment label per aspect: positive, negative, mixed, neutral, not mentioned, or wished for.

Scoring recipe

def compute_f1(predictions, golds):
    tp = sum(1 for p, g in zip(predictions, golds) if p == g)
    fp = sum(1 for p, g in zip(predictions, golds) if p != g and p in golds)
    fn = sum(1 for p, g in zip(predictions, golds) if p != g and g not in predictions)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Custom labels 'not mentioned' and 'wished for' are often missed by models trained on standard sentiment datasets.
• Aspect-level granularity requires aligning predictions with specific topics, not just overall review sentiment.

Evidence (verbatim from paper)

We compute the F1-score Van Rijsbergen ([1979])* between prediction and ground truth for evaluation.

Citation

@misc{billa2025travelbench,
  title={TravelBench : Exploring LLM Performance in Low-Resource Domains},
  author={Billa et al. (2025)},
  year={2025},
  note={arXiv:2510.02719}
}

• arXiv: 2510.02719