Texture Sam Eval

texture-sam-eval

TextureSAM: Towards a Texture Aware Foundation Model for Segmentation — Cohen et al. (2025) (arXiv:2505.16540, 2025)

What this evaluates

Evaluates a model's ability to perform texture-aware segmentation by measuring how well it segments regions based on repeating texture patterns rather than semantic shape cues. It tests generalization on both synthetic texture-only images and natural images, while also checking for catastrophic forgetting on standard semantic benchmarks.

Datasets

• RWTD — total ?; splits: test (-1)
• STMD — total ?; splits: test (-1)
• ADE20K — total ?; splits: val (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union across all classes. Computed as the average of IoU (intersection area divided by union area) for each semantic class.
• ARI — range: [0, 1]
  • Adjusted Rand Index, a clustering similarity metric that measures the agreement between predicted segmentation labels and ground truth labels, adjusted for chance.
• mIoU, Aggr. — range: [0, 1]
  • Aggregated mIoU. Predicted masks are first grouped based on their overlap with ground truth regions to consolidate fragments, then mIoU is computed on the aggregated masks.

Input / output format

Input: RGB image (natural or synthetic texture pattern)

Output: Set of predicted segmentation masks (pixel-wise labels or instance masks)

Scoring recipe

def compute_metrics(pred_masks, gt_masks):
    ious = []
    for class_id in unique_classes:
        pred = (pred_masks == class_id)
        gt = (gt_masks == class_id)
        inter = np.sum(pred & gt)
        union = np.sum(pred | gt)
        ious.append(inter / union if union > 0 else 1.0)
    miou = np.mean(ious)
    ari = adjusted_rand_score(gt_masks.flatten(), pred_masks.flatten())
    agg_pred = aggregate_masks(pred_masks, gt_masks)
    agg_ious = []
    for class_id in unique_classes:
        pred = (agg_pred == class_id)
        gt = (gt_masks == class_id)
        inter = np.sum(pred & gt)
        union = np.sum(pred | gt)
        agg_ious.append(inter / union if union > 0 else 1.0)
    miou_aggr = np.mean(agg_ious)
    return miou, ari, miou_aggr

Common pitfalls

• Over-segmentation by shape-biased models (e.g., SAM-2) artificially lowers mIoU/ARI on texture-only data unless mask aggregation is applied.
• Inference parameters drastically affect results on texture-dominant datasets; using default parameters on SAM-2 yields poor performance compared to modified parameters.
• Aggregation step is essential for fair comparison on texture-defined regions; omitting it penalizes models that correctly identify regions but fragment them into multiple masks.

Evidence (verbatim from paper)

Evaluation is performed using two primary metrics: mean Intersection over Union (mIoU) and Adjusted Rand Index (ARI). To evaluate the overall segmentation quality of TextureSAM vs. the original SAM2 model, we apply mask aggregation, where predicted segmentation masks are grouped based on their overlap with ground truth regions.

Citation

@misc{cohen2025texturesam,
  title={TextureSAM: Towards a Texture Aware Foundation Model for Segmentation},
  author={Cohen et al. (2025)},
  year={2025},
  note={arXiv:2505.16540}
}

• arXiv: 2505.16540