Pqa Biochem Lite Eval

pqa-biochem-lite-eval

PQA: Zero-shot Protein Question Answering for Free-form Scientific Enquiry with Large Language Models — Carrami et al. (2024) (arXiv:2402.13653, 2024)

What this evaluates

Evaluates a model's ability to answer free-form scientific questions about unseen protein sequences using zero-shot multimodal reasoning. It probes biochemical property extraction, functional annotation, and cross-modal alignment between protein embeddings and natural language.

Datasets

• Pika-DS — total 257167; splits: (unstated)

Metrics

• mw MALE (primary) — range: other (lower is better)
  • Mean absolute log error: 1/N * sum(|log10(predicted_MW) / ground_truth_MW|) as written in the paper.
• exact cofactor — range: [0, 1]
  • Exact match score: 1/N * sum(1 if any word in the model's response matches any word in the ground truth cofactor set, else 0).
• location F1 — range: [0, 1]
  • F1 score for sub-cellular location (membrane, nucleus, mitochondrion). Precision and recall are calculated based on exclusive presence of correct labels; responses with multiple or no labels are scored as 'none'.
• is_enzyme F1 — range: [0, 1]
  • F1 score for binary enzyme classification, computed identically to location F1.

Input / output format

Input: A protein amino acid sequence paired with a free-form scientific question.

Output: An open-ended natural language response containing the answer or relevant entities.

Scoring recipe

def score_biochem_lite(preds, gts):
    male_scores, exact_scores, f1_scores = [], [], []
    for p, g in zip(preds, gts):
        male_scores.append(abs(log10(p['mw']) / g['mw']))
        pred_w = set(p['cofactor'].split())
        gt_w = set(g['cofactor'].split())
        exact_scores.append(1 if pred_w & gt_w else 0)
        pred_labels = extract_labels(p['location'])
        gt_labels = set(g['location'])
        prec = len(pred_labels & gt_labels) / max(len(pred_labels), 1)
        rec = len(pred_labels & gt_labels) / max(len(gt_labels), 1)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return {'mw MALE': mean(male_scores), 'exact cofactor': mean(exact_scores), 'location F1': mean(f1_scores)}

Common pitfalls

• Using BLEU/ROUGE instead of entity extraction, as these linguistic metrics poorly correlate with scientific correctness.
• Assuming exact string matching suffices; the protocol requires rule-based entity extraction from open-ended responses.
• For location F1, failing to penalize responses containing multiple sub-cellular labels, which are scored as 'none'.

Evidence (verbatim from paper)

Conventional linguistic metrics like BLEU and ROUGE, while useful in general linguistic contexts, often fall short in assessing scientific correctness and show poor correlation with human judgment (Mathur et al., 2020). As a result these metrics are inadequate for assessing the performance of multimododal PQA models. Therefore, going beyond standard linguistic evaluations, we designed a purpose-built benchmarking approach incorporating a set of predefined, biochemically-significant questions... Biochem-Lite: Pika’s light-weight benchmarking involves a set of pre-defined, scientifically relevant questions with simple answers extracted for each protein using GPT3.5 during Pika-DS creation. ... This response is then processed for rule-based entity extraction and scoring using adequate metrics. For mw MALE, the mean-absolute log error (MALE) of the predicted molecular weight (MW) is: |log10(MW_hat)/(MW)|

Citation

@misc{carrami2024pqa,
  title={PQA: Zero-shot Protein Question Answering for Free-form Scientific Enquiry with Large Language Models},
  author={Carrami et al. (2024)},
  year={2024},
  note={arXiv:2402.13653}
}

• arXiv: 2402.13653