Sentence Level Cal Eval

sentence-level-cal-eval

Evaluating Sentence-Level Relevance Feedback for High-Recall Information Retrieval — Zhang et al. (2018) (arXiv:1803.08988, 2018)

What this evaluates

Evaluates the recall and efficiency of continuous active learning systems for information retrieval when using sentence-level versus document-level relevance feedback. It measures how quickly a simulated reviewer can identify all relevant documents under varying effort models that account for assessment count and sentence reading time.

Datasets

• TREC Total Recall 2015 Track — total ?; splits: test (-1)
• HARD 2004 Track — total ?; splits: test (-1)

Metrics

• Recall@E (primary) — range: [0, 1]
  • Recall at effort E, calculated as the number of relevant documents presented in the system output truncated at effort E, divided by the total number of relevant documents. Effort E can be measured as assessments only (E_judge), sentences read only (E_sent), or a weighted combination E_λ = (1-λ)E_judge + λE_sent. Specific effort levels tested are a*R+b where R is total relevant documents, a∈{1,2,4}, and b∈{0,100,1000}.

Input / output format

Input: System-selected sequence of documents or sentences to be assessed, alongside gold relevance assessments (qrels) for both documents and sentences.

Output: Truncated system output O@E at a specified effort level E, representing the sequence of documents presented to the simulated reviewer up to that effort.

Scoring recipe

def compute_recall_at_effort(system_output, qrels, effort_E, effort_type):
    # Truncate output based on effort type
    if effort_type == 'E_judge':
        truncated_docs = system_output[:effort_E]
    elif effort_type == 'E_sent':
        truncated_docs = get_docs_up_to_sent_count(system_output, effort_E)
    else:  # E_lambda
        truncated_docs = get_docs_up_to_lambda_effort(system_output, effort_E)
    # Compute recall
    relevant_docs = set(qrels.keys())
    presented_docs = set(truncated_docs)
    recall = len(presented_docs & relevant_docs) / len(relevant_docs)
    return recall

Common pitfalls

• Effort is not simply the number of documents reviewed; it depends on whether the system uses sentence-level or document-level feedback, which changes how E is calculated.
• The paper tests specific effort levels defined as a*R+b (where R is total relevant docs, a∈{1,2,4}, b∈{0,100,1000}), not arbitrary fixed numbers.
• For document-level feedback, effort includes reading sentences sequentially until the first relevant one is found, or all sentences if non-relevant, which directly affects E_sent and E_λ.

Evidence (verbatim from paper)

Recall is the number of relevant documents presented to the reviewer for assessment, as a fraction of the total number of relevant documents (R), regardless of whether document- or sentence-level feedback is employed. In our paper, the documents presented to the reviewer are recorded by the system output (O). We measure the recall at effort (Recall@E) using the Equation: Recall@E = |O@E ∩ Relevant documents| / |Relevant documents|

Citation

@misc{zhang2018evaluating,
  title={Evaluating Sentence-Level Relevance Feedback for High-Recall Information Retrieval},
  author={Zhang et al. (2018)},
  year={2018},
  note={arXiv:1803.08988}
}

• arXiv: 1803.08988