Tsynth Detection Eval

tsynth-detection-eval

T-SYNTH: A Knowledge-Based Dataset of Synthetic Breast Images — Wiedeman et al. (2025) (arXiv:2507.04038, 2025)

What this evaluates

Evaluates lesion detection performance on synthetic and real breast mammography/tomosynthesis images. Probes the model's ability to localize lesions across varying breast densities, lesion sizes, and lesion densities using a free-response receiver operating characteristic (FROC) framework.

Datasets

• T-SYNTH — total 4500; splits: train (3000), val (750), test (750); repo https://github.com/DIDSR/tsynth-release
• EMBED — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• FROC (Sensitivity vs. Average False Positives per Image) (primary) — range: [0, 1]
  • Sensitivity (True Positive Rate) is plotted against the average number of false positives per image across varying confidence thresholds. A true positive is scored if the distance between the predicted and ground truth box centers is less than half the ground truth box diagonal or 100 pixels, whichever is smaller.

Input / output format

Input: 2D breast mammography (DM) or tomosynthesis (DBT C-View) images with pixel-level ground truth bounding boxes for lesions.

Output: Predicted bounding boxes with associated confidence scores.

Scoring recipe

def compute_froc(preds, gts, num_images):
    tps, fps, fns = 0, 0, 0
    for p, g in zip(preds, gts):
        matched = set()
        for box in p:
            best_dist, best_idx = float('inf'), -1
            for i, gt in enumerate(g):
                d = center_dist(box, gt)
                if d < best_dist: best_dist, best_idx = d, i
            if best_dist < min(0.5 * g[best_idx].diag, 100):
                tps += 1; matched.add(best_idx)
            else: fps += 1
        fns += len(g) - len(matched)
    sens = tps / (tps + fns)
    avg_fpi = fps / num_images
    return sens, avg_fpi

Common pitfalls

• Using standard precision-recall curves instead of FROC, which is standard for medical screening where negatives dominate.
• Mismatching the true positive matching criterion: the paper uses a specific distance threshold (min(0.5*diagonal, 100px)) rather than IoU.
• Not averaging across multiple random seeds (5 runs) to report mean/min/max sensitivity, leading to high variance in results.

Evidence (verbatim from paper)

Each model is evaluated by calculating the number of true positives (successfully detected lesions), false positives (regions falsely declared as a lesion), and false negatives (missed lesions) that a model outputs over the test set at varying decision thresholds. Similar to the DBTex detection challenge for DBT (Konz et al., [2023]), a true positive was scored if the distance between the prediction and a ground truth boxes’ centers was less than either half of the diagonal length of the ground truth box or 100 pixels. Also similar to this challenge, model sensitivity is reported in relation to the average number of false positives per image as a free-response receiver operator characteristic (FROC) curve.

Citation

@misc{wiedeman2025tsynth,
  title={T-SYNTH: A Knowledge-Based Dataset of Synthetic Breast Images},
  author={Wiedeman et al. (2025)},
  year={2025},
  note={arXiv:2507.04038}
}

• arXiv: 2507.04038