# Pulselm Eval

> This benchmark evaluates multimodal physiological reasoning by testing whether large language models can accurately answer closed-ended questions conditioned on raw photoplethysmography (PPG) waveforms. It probes the model's ability to align continuous biosignal representations with natural language queries across diverse physiological domains and assesses cross-dataset generalization beyond the training distribution. Use when the user wants to benchmark on PulseLM, or asks about evaluating this task. Reports exact-match (EM) accuracy.

- Skill: `qhjqhj00/pulselm-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/pulselm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/pulselm-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/pulselm-eval

---


# pulselm-eval

> PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning — Pham et al. (2026) (arXiv:2603.03331, 2026)

## What this evaluates

This benchmark evaluates multimodal physiological reasoning by testing whether large language models can accurately answer closed-ended questions conditioned on raw photoplethysmography (PPG) waveforms. It probes the model's ability to align continuous biosignal representations with natural language queries across diverse physiological domains and assesses cross-dataset generalization beyond the training distribution.

## Datasets

- **PulseLM** — total 3150000; splits: train (-1), val (-1), test (-1); repo https://github.com/manhph2211/PulseLM

## Metrics

- `exact-match (EM) accuracy` **(primary)** — range: [0, 1]
  - Strict exact-match classification accuracy, computed by checking if the model's predicted answer exactly matches the ground truth label. Scores are averaged across question paraphrases to reduce sensitivity to surface-level linguistic variation.

## Input / output format

**Input**: A 10-second raw PPG waveform segment and a single natural language question (closed-ended QA prompt).

**Output**: A single answer selected from a predefined candidate set.

## Scoring recipe

```python
def compute_em_accuracy(predictions, gold_labels):
    # predictions and gold_labels are lists of strings
    exact_matches = [1.0 if pred == gold else 0.0 for pred, gold in zip(predictions, gold_labels)]
    return sum(exact_matches) / len(exact_matches)
# Note: Final score is averaged across multiple question paraphrases per physiological label.
```

## Common pitfalls

- Models are evaluated using strict exact-match accuracy, not fuzzy or token-level overlap metrics, so minor formatting or casing differences cause a score of 0.
- Cross-dataset generalization is evaluated by training exclusively on VitalDB and testing on unseen datasets (BCG, PPGBP, Sensors, UCI), rather than using standard held-out splits from the same distribution.
- Performance is averaged across multiple question paraphrases for the same label, which can mask variance in how different phrasings affect model reasoning.

## Evidence (verbatim from paper)

> Performance is measured primarily using a strict exact-match classification accuracy, averaged across question paraphrases to reduce sensitivity to surface-level linguistic variation.

## Citation

```bibtex
@misc{pham2026pulselm,
  title={PulseLM: A Foundation Dataset and Benchmark for PPG-Text Learning},
  author={Pham et al. (2026)},
  year={2026},
  note={arXiv:2603.03331}
}
```

- arXiv: 2603.03331