Rp 1k Eval

rp-1k-eval

Novel 3D Scene Understanding Applications From Recurrence in a Single Image — Zhang et al. (2022) (arXiv:2210.07991, 2022)

What this evaluates

Evaluates a model's ability to discover recurring visual patterns in a single image. It measures detection accuracy at both the individual pattern instance level and the whole pattern level against human annotations.

Datasets

• RP-1K — total 1024; splits: test (1024)

Metrics

• RP Instance Precision — range: [0, 1]
  • Precision at the instance level: $P_I = |\mathbf{RPI_A}| / |\mathbf{RPI_D}|$, where $\mathbf{RPI_A}$ are detected instances accepted via IOD > 0.5, and $\mathbf{RPI_D}$ are all detected instances.
• RP Instance Recall (primary) — range: [0, 1]
  • Recall at the instance level: $R_I = |\mathbf{RPI_A}| / |\mathbf{RPI_{GT}}|$, where $\mathbf{RPI_{GT}}$ are all ground truth instances.
• RP Precision — range: [0, 1]
  • Precision at the pattern level: $P_{RP} = |\mathbf{RP_A}| / |\mathbf{RP_D}|$, where $\mathbf{RP_A}$ are detected patterns assigned to a ground truth pattern based on highest instance precision.
• RP Recall — range: [0, 1]
  • Recall at the pattern level: $R_{RP} = |\mathbf{RP_A}| / |\mathbf{RP_{GT}}|$, where $\mathbf{RP_{GT}}$ are all ground truth patterns.

Input / output format

Input: Single RGB image.

Output: A set of detected Recurring Pattern Instances (RPIs) and a set of detected Recurring Patterns (RPs) grouping these instances.

Scoring recipe

def compute_metrics(pred_rpis, gt_rpis, h=0.5):
    accepted = []
    for p in pred_rpis:
        for g in gt_rpis:
            if intersection_area(p, g) / area(p) > h:
                accepted.append(p)
                break
    p_inst = len(accepted) / len(pred_rpis) if pred_rpis else 0
    r_inst = len(accepted) / len(gt_rpis) if gt_rpis else 0
    return p_inst, r_inst

Common pitfalls

• Uses Intersection-over-Detection (IOD) instead of standard IoU to evaluate partial overlaps of recurring patterns.
• RP-level matching assigns a detected RP to the ground truth RP with the highest instance-level precision, not IoU or recall.
• The IOD threshold h=0.5 is fixed for reported results; altering it changes acceptance rates significantly.

Evidence (verbatim from paper)

We propose an intersection-over-detection (IOD) metric as follows. If $RPI_{i}$ is a detected RPI, and $RPI_{GT_{j}}$ a groundtruth RPI. $RPI_{j}$ is considered acceptable if and only if $(RPI_{i}\cap RPI_{GT_{j}})/RPI_{i}>h$, where $h$ is a numerical threshold. Given a detected RP with a set of RPIs $\mathbf{RPI_{D}}$, and some ground truth RP with a set of ground truth RPIs $\mathbf{RPI_{GT}}$, and the set of acceptable RPIs denoted as $\mathbf{RPI_{A}}$, RP Instance level precision $P_{I}$ and recall $R_{I}$ rates are defined as: $P_{I}=|\mathbf{RPI_{A}}|/|\mathbf{RPI_{D}}|$, $R_{I}=|\mathbf{RPI_{A}}|/|\mathbf{RPI_{GT}}|$

Citation

@misc{zhang2022novel3d,
  title={Novel 3D Scene Understanding Applications From Recurrence in a Single Image},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2210.07991}
}

• arXiv: 2210.07991