Tampar Eval

tampar-eval

TAMPAR: Visual Tampering Detection for Parcel Logistics in Postal Supply Chains — Alexander Naumann et al. (arXiv:2311.03124, 2023)

What this evaluates

This benchmark probes a model's ability to detect visual tampering on parcel logistics items by comparing a single RGB image to a reference database. It evaluates the pipeline's robustness in detecting corner keypoints, performing perspective transformation to generate viewpoint-invariant views, and accurately identifying appearance changes across varying angles, lighting, and lens distortions.

Datasets

• TAMPAR — total 3306; splits: test (3306)

Metrics

• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Thresholds are determined via a depth-one decision tree trained on similarity metrics.
• Keypoint AP — range: [0, 1]
  • Average Precision for keypoint detection, computed with annotation precision thresholds κ_i (0.05 for visible corners, 0.1 for self-occluded corners).

Input / output format

Input: Single RGB image of a parcel to be inspected, compared against a reference image of the same parcel.

Output: Binary tampering label (tampered/not tampered) and/or a binary change map, along with 8 corner keypoint coordinates.

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Models are trained exclusively on synthetic data (Parcel3D) but evaluated on real-world datasets, causing a domain gap that lowers performance.
• Ground truth keypoints are derived from 3D bounding boxes, causing frequent misalignment with actual parcel corners, especially for damaged parcels.
• Strong lens distortions can push keypoints outside the image boundaries or corrupt annotations, degrading detection accuracy.

Evidence (verbatim from paper)

Results in Tab. 2 using predicted keypoints show that (SimSaC, LPIPS) yields the best performance and reaches 0.81 accuracy and an F1-Score of 0.83.

Citation

@misc{naumann2023tampar,
  title={TAMPAR: Visual Tampering Detection for Parcel Logistics in Postal Supply Chains},
  author={Alexander Naumann et al.},
  year={2023},
  note={arXiv:2311.03124}
}

• arXiv: 2311.03124