Q Measure

q-measure

Graded Relevance Assessments and Graded Relevance Measures of NTCIR: A Survey of the First Twenty Years — Sakai et al. (2019) (arXiv:1903.11272, 2019)

What this evaluates

Evaluates ranked retrieval lists using graded relevance assessments, balancing precision and cumulative gain while penalizing lower-ranked relevant documents.

Datasets

• (no dataset; pure metric skill)

Metrics

• Q-measure (primary) — range: [0, 1]
  • Q = (1/R) * sum_{r} I(r) * BR(r), where BR(r) = (C(r) + beta * cg(r)) / (r + beta * cg*(r)). C(r) is binary precision count, cg(r) is cumulative graded gain, cg*(r) is cumulative gain of ideal list, R is total relevant docs, beta is patience (usually 1). Reduces to AP when beta=0.

Input / output format

Input: A ranked list of retrieved documents/items for a query, along with graded relevance labels (e.g., 0=nonrelevant, 1=partially relevant, 2=highly relevant) for each item.

Output: A single scalar score in [0, 1] representing the retrieval quality.

Scoring recipe

def q_measure(retrieved_docs, relevance_labels, beta=1.0):
    R = sum(1 for x in relevance_labels if x > 0)
    if R == 0: return 0.0
    gains = [gain_value[x] for x in relevance_labels]
    ideal_gains = sorted(gains, reverse=True)
    C = 0; cg = 0; cg_star = 0
    Q = 0.0
    for r, rel in enumerate(relevance_labels, 1):
        I_r = 1 if rel > 0 else 0
        C += I_r
        cg += gains[r-1]
        cg_star += ideal_gains[r-1]
        BR = (C + beta * cg) / (r + beta * cg_star)
        Q += I_r * BR
    return Q / R

Common pitfalls

• Equivalence classes for answer strings (e.g., in QA tasks) are handled by counting only one relevant string per class, unlike standard document retrieval.
• The patience parameter beta is usually set to 1 but controls the trade-off between precision and nCG; setting beta=0 reduces Q to AP.
• Q and nDCG behave similarly but use different mechanisms to penalize low-rank relevant documents (denominator rank vs. logarithmic discount).

Evidence (verbatim from paper)

The Q-measure is defined as Q = (1/R) sum_{r} I(r) BR(r), where BR(r) is the blended ratio given by BR(r) = (C(r) + beta cg(r)) / (r + beta cg*(r)). Here, beta is the patience parameter which is usually set to one; its significance is discussed in Sakai (2014). Note that C(r)/r represents binary Precision at rank r; hence, both precision and nCG are embedded in Eq. 2.

Citation

@misc{sakai2019graded,
  title={Graded Relevance Assessments and Graded Relevance Measures of NTCIR: A Survey of the First Twenty Years},
  author={Sakai et al. (2019)},
  year={2019},
  note={arXiv:1903.11272}
}

• arXiv: 1903.11272