# Pqa Biochem Lite Eval

> Evaluates a model's ability to answer free-form scientific questions about unseen protein sequences using zero-shot multimodal reasoning. It probes biochemical property extraction, functional annotation, and cross-modal alignment between protein embeddings and natural language. Use when the user wants to benchmark on Pika-DS, or asks about evaluating this task. Reports mw MALE.

- Skill: `qhjqhj00/pqa-biochem-lite-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/pqa-biochem-lite-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/pqa-biochem-lite-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/pqa-biochem-lite-eval

---


# pqa-biochem-lite-eval

> PQA: Zero-shot Protein Question Answering for Free-form Scientific Enquiry with Large Language Models — Carrami et al. (2024) (arXiv:2402.13653, 2024)

## What this evaluates

Evaluates a model's ability to answer free-form scientific questions about unseen protein sequences using zero-shot multimodal reasoning. It probes biochemical property extraction, functional annotation, and cross-modal alignment between protein embeddings and natural language.

## Datasets

- **Pika-DS** — total 257167; splits: (unstated)

## Metrics

- `mw MALE` **(primary)** — range: other (lower is better)
  - Mean absolute log error: 1/N * sum(|log10(predicted_MW) / ground_truth_MW|) as written in the paper.
- `exact cofactor` — range: [0, 1]
  - Exact match score: 1/N * sum(1 if any word in the model's response matches any word in the ground truth cofactor set, else 0).
- `location F1` — range: [0, 1]
  - F1 score for sub-cellular location (membrane, nucleus, mitochondrion). Precision and recall are calculated based on exclusive presence of correct labels; responses with multiple or no labels are scored as 'none'.
- `is_enzyme F1` — range: [0, 1]
  - F1 score for binary enzyme classification, computed identically to location F1.

## Input / output format

**Input**: A protein amino acid sequence paired with a free-form scientific question.

**Output**: An open-ended natural language response containing the answer or relevant entities.

## Scoring recipe

```python
def score_biochem_lite(preds, gts):
    male_scores, exact_scores, f1_scores = [], [], []
    for p, g in zip(preds, gts):
        male_scores.append(abs(log10(p['mw']) / g['mw']))
        pred_w = set(p['cofactor'].split())
        gt_w = set(g['cofactor'].split())
        exact_scores.append(1 if pred_w & gt_w else 0)
        pred_labels = extract_labels(p['location'])
        gt_labels = set(g['location'])
        prec = len(pred_labels & gt_labels) / max(len(pred_labels), 1)
        rec = len(pred_labels & gt_labels) / max(len(gt_labels), 1)
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return {'mw MALE': mean(male_scores), 'exact cofactor': mean(exact_scores), 'location F1': mean(f1_scores)}
```

## Common pitfalls

- Using BLEU/ROUGE instead of entity extraction, as these linguistic metrics poorly correlate with scientific correctness.
- Assuming exact string matching suffices; the protocol requires rule-based entity extraction from open-ended responses.
- For location F1, failing to penalize responses containing multiple sub-cellular labels, which are scored as 'none'.

## Evidence (verbatim from paper)

> Conventional linguistic metrics like BLEU and ROUGE, while useful in general linguistic contexts, often fall short in assessing scientific correctness and show poor correlation with human judgment (Mathur et al., 2020). As a result these metrics are inadequate for assessing the performance of multimododal PQA models. Therefore, going beyond standard linguistic evaluations, we designed a purpose-built benchmarking approach incorporating a set of predefined, biochemically-significant questions... Biochem-Lite: Pika’s light-weight benchmarking involves a set of pre-defined, scientifically relevant questions with simple answers extracted for each protein using GPT3.5 during Pika-DS creation. ... This response is then processed for rule-based entity extraction and scoring using adequate metrics. For mw MALE, the mean-absolute log error (MALE) of the predicted molecular weight (MW) is: |log10(MW_hat)/(MW)|

## Citation

```bibtex
@misc{carrami2024pqa,
  title={PQA: Zero-shot Protein Question Answering for Free-form Scientific Enquiry with Large Language Models},
  author={Carrami et al. (2024)},
  year={2024},
  note={arXiv:2402.13653}
}
```

- arXiv: 2402.13653