# Reta Benchmark Eval

> Evaluates the accuracy and structural consistency of retinal vascular tree annotations across pixel, vessel segment, and network levels, ensuring topological correctness and geometrical plausibility. Use when the user wants to benchmark on RETA Benchmark, or asks about evaluating this task. Reports multi_stage_annotation.

- Skill: `qhjqhj00/reta-benchmark-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/reta-benchmark-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/reta-benchmark-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/reta-benchmark-eval

---


# reta-benchmark-eval

> The RETA Benchmark for Retinal Vascular Tree Analysis — Lyu et al. (2021) (arXiv:2111.11658, 2021)

## What this evaluates

Evaluates the accuracy and structural consistency of retinal vascular tree annotations across pixel, vessel segment, and network levels, ensuring topological correctness and geometrical plausibility.

## Datasets

- **RETA Benchmark** — total 81; splits: train (54), test (27)

## Metrics

- `multi_stage_annotation` **(primary)** — range: other
  - Expert-driven validation across three stages: (1) pixel-level binary mask correction, (2) structure-level artery/vein classification, and (3) network-level topological validation. Topological rules require bifurcation degree ≤ 3 and no graph cycles (verified via minimum spanning tree comparison). Geometrical rules flag abnormalities if focal diameter variation >3px, parent calibre < child calibre, bifurcation angle >150°, or tortuosity index >5.

## Input / output format

**Input**: Resized color fundus images (1024×1024) with black background, optionally enhanced via CLAHE or LCE. Provided as PNG images and MATLAB .mat files containing original/enhanced images, FOV masks, and annotation structures.

**Output**: Binary vessel masks, artery/vein segmentation masks, 1-pixel width skeletons, bifurcation/terminal landmark points, vascular tree graphs, and thick/thin vessel masks with abnormal location labels.

## Scoring recipe

```python
def evaluate_annotation(pred_mask, pred_av, pred_skeleton, pred_graph, gold_mask, gold_av):
    # Stage 1: Pixel-level accuracy
    pixel_acc = compute_iou(pred_mask, gold_mask)
    # Stage 2: Structure-level A/V consistency
    av_acc = compute_pixel_accuracy(pred_av, gold_av)
    # Stage 3: Topological validation
    topo_valid = (max_degree(pred_graph) <= 3) and (not has_cycles(pred_graph, method='MST'))
    # Stage 4: Geometrical validation
    geo_valid = (max_diameter_variation(pred_skeleton) <= 3) and \
                (all(parent_calibre >= child_calibre) for edges in pred_graph) and \
                (max_bifurcation_angle(pred_graph) <= 150) and \
                (max_tortuosity_index(pred_skeleton) <= 5)
    return {'pixel_acc': pixel_acc, 'av_acc': av_acc, 'topo_valid': topo_valid, 'geo_valid': geo_valid}
```

## Common pitfalls

- Down-sampling to 1024×1024 causes loss of tiny lesions (<15 pixels), specifically 2/3,497 microaneurysms and 18/11,642 hard exudates.
- A/V classification strictly uses three classes (artery, vein, crossing) without an 'uncertain' category, unlike RITE, LES-AV, and HRF-AV.
- Skeletonization fails at image borders where partial vessel cross-sections fall outside the frame, requiring manual correction.
- Graph construction explicitly excludes the optic disk (OD) region to prevent unreliable tangled vessel graphs.

## Evidence (verbatim from paper)

> The proposed method is able to control inter-annotator variability and intra-annotator variability because trained annotator performs on multi-stage annotation and subsequent label disambiguation. Accuracy of arterial/venous pixel-level annotation is of greater importance because it could ensure reliable annotations of vessel skeletons, bifurcation and trees shown in Figure 1. Also, a completed and reliable binary vessel masks depends on annotation quality of arterial and venous vessels.

## Citation

```bibtex
@misc{lyu2021reta,
  title={The RETA Benchmark for Retinal Vascular Tree Analysis},
  author={Lyu et al. (2021)},
  year={2021},
  note={arXiv:2111.11658}
}
```

- arXiv: 2111.11658