Siqa Eval

siqa-eval

SIQA: Toward Reliable Scientific Image Quality Assessment — Li et al. (2026) (arXiv:2603.06700, 2026)

What this evaluates

Evaluates multimodal large language models on their ability to understand and assess the quality of scientific images. It probes two distinct capabilities: factual reasoning about scientific content (SIQA-U) and alignment with human expert judgments on perceptual and knowledge dimensions (SIQA-S).

Datasets

• SIQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted multiple-choice options (A–D) across four scientific quality dimensions and three question types.
• SRCC (primary) — range: [-1, 1]
  • Spearman Rank Correlation Coefficient measuring the monotonic relationship between predicted scores and human ratings.
• PLCC — range: [-1, 1]
  • Pearson Linear Correlation Coefficient measuring the linear relationship between predicted scores and human ratings.

Input / output format

Input: For SIQA-U: an image paired with a multiple-choice question (options A–D). For SIQA-S: an image paired with a prompt requesting a quality rating across five levels (Bad, Poor, Fair, Good, Excellent).

Output: For SIQA-U: a single character (A, B, C, or D), with extra text ignored if the answer is clearly identifiable. For SIQA-S: logits for the five quality tokens, used to compute a continuous predicted score.

Scoring recipe

# SIQA-U
acc = sum(1 for pred, gold in zip(predictions, golds) if pred.strip()[0] == gold) / len(golds)

# SIQA-S
scores_pred = []
for logits in predictions:
    probs = softmax(logits)
    levels = [1, 2, 3, 4, 5]
    scores_pred.append(sum(p * l for p, l in zip(probs, levels)))
srcc = spearmanr(scores_pred, golds).correlation
plcc = pearsonr(scores_pred, golds).correlation

Common pitfalls

• Models may output verbose text; the protocol strictly requires extracting a clearly identifiable single character (A–D), otherwise marking it incorrect.
• SIQA-S evaluation is restricted to models with publicly available weights or API logprob support, as it requires access to token-level logits for the five rating levels.
• Fine-tuning on SIQA-S can artificially inflate correlation scores through label fitting, masking deficits in actual scientific reasoning measured by SIQA-U.

Evidence (verbatim from paper)

These predicted scores are then used to compute the SRCC and Pearson Linear Correlation Coefficient (PLCC) for both the perception and knowledge dimensions.

Citation

@misc{li2026siqa,
  title={SIQA: Toward Reliable Scientific Image Quality Assessment},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.06700}
}

• arXiv: 2603.06700