Reta Benchmark Eval

reta-benchmark-eval

The RETA Benchmark for Retinal Vascular Tree Analysis — Lyu et al. (2021) (arXiv:2111.11658, 2021)

What this evaluates

Evaluates the accuracy and structural consistency of retinal vascular tree annotations across pixel, vessel segment, and network levels, ensuring topological correctness and geometrical plausibility.

Datasets

• RETA Benchmark — total 81; splits: train (54), test (27)

Metrics

• multi_stage_annotation (primary) — range: other
  • Expert-driven validation across three stages: (1) pixel-level binary mask correction, (2) structure-level artery/vein classification, and (3) network-level topological validation. Topological rules require bifurcation degree ≤ 3 and no graph cycles (verified via minimum spanning tree comparison). Geometrical rules flag abnormalities if focal diameter variation >3px, parent calibre < child calibre, bifurcation angle >150°, or tortuosity index >5.

Input / output format

Input: Resized color fundus images (1024×1024) with black background, optionally enhanced via CLAHE or LCE. Provided as PNG images and MATLAB .mat files containing original/enhanced images, FOV masks, and annotation structures.

Output: Binary vessel masks, artery/vein segmentation masks, 1-pixel width skeletons, bifurcation/terminal landmark points, vascular tree graphs, and thick/thin vessel masks with abnormal location labels.

Scoring recipe

def evaluate_annotation(pred_mask, pred_av, pred_skeleton, pred_graph, gold_mask, gold_av):
    # Stage 1: Pixel-level accuracy
    pixel_acc = compute_iou(pred_mask, gold_mask)
    # Stage 2: Structure-level A/V consistency
    av_acc = compute_pixel_accuracy(pred_av, gold_av)
    # Stage 3: Topological validation
    topo_valid = (max_degree(pred_graph) <= 3) and (not has_cycles(pred_graph, method='MST'))
    # Stage 4: Geometrical validation
    geo_valid = (max_diameter_variation(pred_skeleton) <= 3) and \
                (all(parent_calibre >= child_calibre) for edges in pred_graph) and \
                (max_bifurcation_angle(pred_graph) <= 150) and \
                (max_tortuosity_index(pred_skeleton) <= 5)
    return {'pixel_acc': pixel_acc, 'av_acc': av_acc, 'topo_valid': topo_valid, 'geo_valid': geo_valid}

Common pitfalls

• Down-sampling to 1024×1024 causes loss of tiny lesions (<15 pixels), specifically 2/3,497 microaneurysms and 18/11,642 hard exudates.
• A/V classification strictly uses three classes (artery, vein, crossing) without an 'uncertain' category, unlike RITE, LES-AV, and HRF-AV.
• Skeletonization fails at image borders where partial vessel cross-sections fall outside the frame, requiring manual correction.
• Graph construction explicitly excludes the optic disk (OD) region to prevent unreliable tangled vessel graphs.

Evidence (verbatim from paper)

The proposed method is able to control inter-annotator variability and intra-annotator variability because trained annotator performs on multi-stage annotation and subsequent label disambiguation. Accuracy of arterial/venous pixel-level annotation is of greater importance because it could ensure reliable annotations of vessel skeletons, bifurcation and trees shown in Figure 1. Also, a completed and reliable binary vessel masks depends on annotation quality of arterial and venous vessels.

Citation

@misc{lyu2021reta,
  title={The RETA Benchmark for Retinal Vascular Tree Analysis},
  author={Lyu et al. (2021)},
  year={2021},
  note={arXiv:2111.11658}
}

• arXiv: 2111.11658