Semeval Question Relevancy Eval

semeval-question-relevancy-eval

A strong baseline for question relevancy ranking — González-Garduño et al. (2018) (arXiv:1808.08836, 2018)

What this evaluates

Evaluates a model's ability to rank question-answer pairs by relevance. It probes the system's capacity to understand semantic similarity and perform information retrieval tasks using language-independent features and multi-task learning.

Datasets

• SemEval-2016 Task 3 — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• MAP (primary) — range: [0, 100]
  • Mean Average Precision; standard information retrieval metric averaging precision at relevant document ranks across queries.
• Accuracy — range: [0, 100]
  • Accuracy; fraction of correctly classified or correctly ranked pairs.

Input / output format

Input: Pairs of questions (or question-answer pairs) to be evaluated for relevance.

Output: Ranked list of pairs by relevance score, or binary/multi-class relevance labels.

Scoring recipe

# MAP calculation
ap_scores = []
for query in queries:
    relevant_docs = gold_relevance[query]
    ranked_docs = model_rank(query)
    hits = 0
    sum_prec = 0
    for i, doc in enumerate(ranked_docs):
        if doc in relevant_docs:
            hits += 1
            sum_prec += hits / (i + 1)
    ap_scores.append(sum_prec / len(relevant_docs))
map_score = sum(ap_scores) / len(ap_scores)
# ACC calculation
acc_score = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

Common pitfalls

• Confusing the primary ranking metric (MAP) with classification accuracy (Accuracy), as both are reported but MAP is the official SemEval metric.
• Assuming full auxiliary datasets (FNC, MultiNLI) are used for training; the paper explicitly samples them to match the size of SemEval's train/dev/test sets.
• Overlooking that the model relies on simple language-independent distance metrics (cosine, Euclidean, Jaccard) rather than complex deep architectures.

Evidence (verbatim from paper)

We present the official metric from the SemEval task, as well as other common metrics. For the SemEval-16 data, our multitask MLP architecture with a question-answer auxiliary task performed best on all metrics, except accuracy, where the multi-task MLP using all auxiliary tasks performed best. We outperform the winning systems of both the SemEval 2016 and 2017 campaigns.

Citation

@misc{gonzalezgarduno2018strong,
  title={A strong baseline for question relevancy ranking},
  author={González-Garduño et al. (2018)},
  year={2018},
  note={arXiv:1808.08836}
}

• arXiv: 1808.08836