medscope-svu-eval
MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling — Li et al. (2026) (arXiv:2602.13332, 2026)
What this evaluates
Evaluates multimodal models on multi-grained video description and fine-grained temporal/perceptual visual reasoning using long-form medical videos.
Datasets
- SVU-31K — total ?; splits: test (-1)
Metrics
CI, DO, CU, TU (primary) — range: other
- Scores for Correctness of Information, Detail Orientation, Contextual Understanding, and Temporal Understanding, assessed via human or LLM judges on a continuous scale.
BLEU-4, CIDEr, METEOR, ROUGE-L — range: percent
- Standard n-gram and weighted overlap metrics for evaluating generated video descriptions and reasoning text against reference captions.
Tool usage rate — range: percent
- Percentage of instances where the model invokes a tool during reasoning.
Input / output format
Input: Long-form medical video paired with a text prompt asking for video description or visual reasoning.
Output: Text response containing video descriptions, reasoning steps, tool calls, and final answers.
Scoring recipe
def compute_text_metrics(preds, refs):
bleu4 = nltk.bleu4(preds, refs)
cider = compute_cider(preds, refs)
meteor = compute_meteor(preds, refs)
rouge_l = compute_rouge(preds, refs, 'rougeL')
return bleu4, cider, meteor, rouge_l
def compute_tool_rate(preds):
return sum(1 for p in preds if 'tool_call' in p) / len(preds)
Common pitfalls
- CI, DO, CU, TU scores are reported as continuous values (e.g., 4.77), likely from LLM-as-judge or human raters on a scale, not standard discrete accuracy.
- Tool usage rate is reported as a percentage but the paper does not specify the exact tool API or how tool calls are parsed from the model's output.
Evidence (verbatim from paper)
On fine-grained reasoning, MedScope sets a new state of the art, reaching 4.56 CIDEr on temporal reasoning and 10.93 CIDEr on perceptual reasoning, with clear gains over the medical baseline SurgVidLM.
Citation
@misc{li2026medscope,
title={MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling},
author={Li et al. (2026)},
year={2026},
note={arXiv:2602.13332}
}
1---2name: medscope-svu-eval3description: Evaluates multimodal models on multi-grained video description and fine-grained temporal/perceptual visual reasoning using long-form medical videos. Use when the user wants to benchmark on SVU-31K, or asks about evaluating this task. Reports CI, DO, CU, TU.4---56# medscope-svu-eval78> MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling — Li et al. (2026) (arXiv:2602.13332, 2026)910## What this evaluates1112Evaluates multimodal models on multi-grained video description and fine-grained temporal/perceptual visual reasoning using long-form medical videos.1314## Datasets1516- **SVU-31K** — total ?; splits: test (-1)1718## Metrics1920- `CI, DO, CU, TU` **(primary)** — range: other21 - Scores for Correctness of Information, Detail Orientation, Contextual Understanding, and Temporal Understanding, assessed via human or LLM judges on a continuous scale.22- `BLEU-4, CIDEr, METEOR, ROUGE-L` — range: percent23 - Standard n-gram and weighted overlap metrics for evaluating generated video descriptions and reasoning text against reference captions.24- `Tool usage rate` — range: percent25 - Percentage of instances where the model invokes a tool during reasoning.2627## Input / output format2829**Input**: Long-form medical video paired with a text prompt asking for video description or visual reasoning.3031**Output**: Text response containing video descriptions, reasoning steps, tool calls, and final answers.3233## Scoring recipe3435```python36def compute_text_metrics(preds, refs):37 bleu4 = nltk.bleu4(preds, refs)38 cider = compute_cider(preds, refs)39 meteor = compute_meteor(preds, refs)40 rouge_l = compute_rouge(preds, refs, 'rougeL')41 return bleu4, cider, meteor, rouge_l4243def compute_tool_rate(preds):44 return sum(1 for p in preds if 'tool_call' in p) / len(preds)45```4647## Common pitfalls4849- CI, DO, CU, TU scores are reported as continuous values (e.g., 4.77), likely from LLM-as-judge or human raters on a scale, not standard discrete accuracy.50- Tool usage rate is reported as a percentage but the paper does not specify the exact tool API or how tool calls are parsed from the model's output.5152## Evidence (verbatim from paper)5354> On fine-grained reasoning, MedScope sets a new state of the art, reaching 4.56 CIDEr on temporal reasoning and 10.93 CIDEr on perceptual reasoning, with clear gains over the medical baseline SurgVidLM.5556## Citation5758```bibtex59@misc{li2026medscope,60 title={MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling},61 author={Li et al. (2026)},62 year={2026},63 note={arXiv:2602.13332}64}65```6667- arXiv: 2602.13332
qhjqhj00