Healthslm Bench Eval

healthslm-bench-eval

HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring — Wang et al. (2025) (arXiv:2509.07260, 2025)

What this evaluates

Evaluates small language models on health prediction tasks using wearable sensor data. It probes the models' ability to infer physiological and mental health states (e.g., stress, fatigue, depression) from temporal behavioral and physiological features under zero-shot, few-shot, and instruction-tuned settings.

Datasets

• PMData — total ?; splits: train (-1), test (-1)
• GLOBEM — total ?; splits: train (-1), test (-1)
• AW-FB — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of the total number of predictions for classification tasks.
• mean absolute error (MAE) — range: other
  • Average of the absolute differences between predicted and actual values for regression tasks.

Input / output format

Input: Temporal sequences of wearable sensor features (steps, calories, resting heart rate, sleep metrics) formatted into 14-day windows, incorporated into query prompts.

Output: Text predictions generated by the SLM, compared against self-reported ground-truth labels (classification categories or regression values).

Scoring recipe

if task_type == 'regression':
    mae = sum(abs(pred - gold) for pred, gold in zip(predictions, golds)) / len(predictions)
    return mae
else:
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    accuracy = correct / len(predictions)
    return accuracy

Common pitfalls

• Class imbalance significantly degrades performance, particularly in few-shot settings, which the authors highlight as a key limitation.
• Efficiency metrics (TTFT, ITPS, OTPS, RAM/CPU) are only evaluated on the top-performing health-adapted SLMs deployed on an iPhone 15 Pro Max, not across all nine models.
• The 14-day window aggregation and fixed 8:2 train/test split are applied uniformly, which may obscure temporal dynamics or require careful handling of participant-level data leakage.

Evidence (verbatim from paper)

To evaluate model performance under zero-shot, few-shot, and instructional-tuning settings, we use mean absolute error (MAE) for regression tasks and accuracy for classification tasks. For efficiency evaluation of mobile deployment, we assess the models latency using metrics such as Time-to-First-Token (TTFT), Input Tokens Per Second (ITPS), Output Tokens Per Second (OTPS), and Output Evaluation Time (OET) and Total Time. In addition, We also track CPU and RAM usage to evaluate on-device resource consumption.

Citation

@misc{wang2025healthslmbench,
  title={HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.07260}
}

• arXiv: 2509.07260