qhjqhj00 aspect-extraction-eval
An Aspect Extraction Framework using Different Embedding Types, Learning Models, and Dependency Structure — Erkan et al. (2025) (arXiv:2503.03512, 2025)
What this evaluates
Evaluates a model's ability to identify aspect terms (targets), their categories, sentiment polarity, and exact character positions within restaurant review sentences in Turkish.
Datasets
- SemEval 2016 Turkish Restaurant Reviews — total 1248; splits: train (1104), test (144); repo https://github.com/alierkan/Turkish-ABSA
- SemEval 2016 English-Translated Restaurant Reviews — total 2676; splits: train (2000), test (676); repo https://github.com/alierkan/Turkish-ABSA
Metrics
F1 score(primary) — range: [0, 1]- Harmonic mean of precision and recall for correctly predicted aspect spans and attributes. Calculated per aspect instance across the test set.
Input / output format
Input: Raw text of a sentence or review.
Output: Structured prediction containing aspect target string, category, polarity, and character span indices (from-to).
Scoring recipe
def compute_f1(predictions, gold):
correct = 0
for pred, gold_item in zip(predictions, gold):
if (pred['target'] == gold_item['target'] and
pred['category'] == gold_item['category'] and
pred['polarity'] == gold_item['polarity'] and
pred['from'] == gold_item['from'] and
pred['to'] == gold_item['to']):
correct += 1
precision = correct / len(predictions) if predictions else 0
recall = correct / len(gold) if gold else 0
return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Translation-induced position shifts require careful relabeling of aspect spans.
- Aspect term normalization is critical to avoid counting identical concepts as different spans.
- XML format requires parsing nested sentence and opinion tags correctly.
Evidence (verbatim from paper)
The SemEval Restaurant Reviews datasets are stored in XML format. Listing [1] shows an example review from the English Restaurant Reviews dataset. A review is formed of a number of sentences. Each sentence includes zero or more aspect/sentiment information which is composed of an aspect ("target"), category of the aspect ("category"), sentiment for the aspect ("polarity"), and the position of the aspect within the sentence ("from"-"to").
Citation
@misc{erkan2025aspect,
title={An Aspect Extraction Framework using Different Embedding Types, Learning Models, and Dependency Structure},
author={Erkan et al. (2025)},
year={2025},
note={arXiv:2503.03512}
}
- arXiv: 2503.03512