Semeval2022 Task10 Eval

semeval2022-task10-eval

LyS_ACoruña at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing — Alonso-Alonso et al. (2022) (arXiv:2204.12820, 2022)

What this evaluates

This benchmark evaluates structured sentiment analysis by testing a model's ability to extract sentiment targets, opinions, and their relational dependencies from text. It probes cross-lingual generalization and the capacity to repurpose semantic dependency parsers for sentiment graph generation.

Datasets

• SemEval-2022 Task 10 — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • F1 = 2 * (precision * recall) / (precision + recall), computed over the set of predicted sentiment graph elements (nodes and edges) against gold annotations.

Input / output format

Input: Raw text sentences or reviews with annotated sentiment targets, opinions, and dependency relations.

Output: Structured sentiment graphs (sentiment dependency parses) representing extracted targets, opinions, and their directed relations.

Scoring recipe

def compute_f1(pred_graph, gold_graph):
    pred_elements = set(pred_graph.nodes + pred_graph.edges)
    gold_elements = set(gold_graph.nodes + gold_graph.edges)
    tp = len(pred_elements & gold_elements)
    fp = len(pred_elements - gold_elements)
    fn = len(gold_elements - pred_elements)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• Evaluating only sentiment polarity or target extraction without capturing the full dependency graph structure will yield artificially high scores and misrepresent model capability.
• The SemEval-2022 organizers updated training files due to segmentation bugs shortly before the deadline; models must be trained on the final updated versions to ensure fair comparison across submissions.
• Cross-lingual zero-shot setups using word-level translated treebanks underperform compared to training on merged English treebanks without translation, contrary to typical cross-lingual expectations.

Evidence (verbatim from paper)

Citation

@misc{alons2022lysacoru,
  title={LyS_ACoruña at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing},
  author={Alonso-Alonso et al. (2022)},
  year={2022},
  note={arXiv:2204.12820}
}

• arXiv: 2204.12820