Pulselm Eval

pulselm-eval

PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning — Pham et al. (2026) (arXiv:2603.03331, 2026)

What this evaluates

This benchmark evaluates multimodal physiological reasoning by testing whether large language models can accurately answer closed-ended questions conditioned on raw photoplethysmography (PPG) waveforms. It probes the model's ability to align continuous biosignal representations with natural language queries across diverse physiological domains and assesses cross-dataset generalization beyond the training distribution.

Datasets

• PulseLM — total 3150000; splits: train (-1), val (-1), test (-1); repo https://github.com/manhph2211/PulseLM

Metrics

• exact-match (EM) accuracy (primary) — range: [0, 1]
  • Strict exact-match classification accuracy, computed by checking if the model's predicted answer exactly matches the ground truth label. Scores are averaged across question paraphrases to reduce sensitivity to surface-level linguistic variation.

Input / output format

Input: A 10-second raw PPG waveform segment and a single natural language question (closed-ended QA prompt).

Output: A single answer selected from a predefined candidate set.

Scoring recipe

def compute_em_accuracy(predictions, gold_labels):
    # predictions and gold_labels are lists of strings
    exact_matches = [1.0 if pred == gold else 0.0 for pred, gold in zip(predictions, gold_labels)]
    return sum(exact_matches) / len(exact_matches)
# Note: Final score is averaged across multiple question paraphrases per physiological label.

Common pitfalls

• Models are evaluated using strict exact-match accuracy, not fuzzy or token-level overlap metrics, so minor formatting or casing differences cause a score of 0.
• Cross-dataset generalization is evaluated by training exclusively on VitalDB and testing on unseen datasets (BCG, PPGBP, Sensors, UCI), rather than using standard held-out splits from the same distribution.
• Performance is averaged across multiple question paraphrases for the same label, which can mask variance in how different phrasings affect model reasoning.

Evidence (verbatim from paper)

Performance is measured primarily using a strict exact-match classification accuracy, averaged across question paraphrases to reduce sensitivity to surface-level linguistic variation.

Citation

@misc{pham2026pulselm,
  title={PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning},
  author={Pham et al. (2026)},
  year={2026},
  note={arXiv:2603.03331}
}

• arXiv: 2603.03331