Poi Eval

poi-eval

Beyond Visual Appearances: Privacy-sensitive Objects Identification via Hybrid Graph Reasoning — Jiang et al. (2024) (arXiv:2406.12736, 2024)

What this evaluates

Probes a model's ability to identify privacy-sensitive objects in images by reasoning about scene context rather than relying solely on visual appearance. It evaluates whether the system can distinguish between obvious privacy leaks (e.g., faces) and context-dependent sensitive information (e.g., people in specific roles).

Datasets

• MOSAIC — total 13384; splits: test (13384)
• PRIVACY1000 — total 1000; splits: train (800), val (200)

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed per dataset split.
• Precision — range: [0, 1]
  • Ratio of true positive detections to all positive detections: Precision = TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of true positive detections to all actual positives: Recall = TP / (TP + FN).

Input / output format

Input: RGB images containing scenes with potential privacy-sensitive objects.

Output: Bounding box coordinates and a binary privacy-sensitive classification label for each detected object.

Scoring recipe

def compute_metrics(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if iou(p.box, g.box) > 0.5 and p.label == g.label)
    fp = len(preds) - tp
    fn = len(gold) - tp
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'precision': prec, 'recall': rec, 'f1_score': f1}

Common pitfalls

• PRIVACY1000 is explicitly stated as not publicly available, hindering direct reproduction.
• Privacy sensitivity is subjective; annotations use a majority-rule approach across multiple annotators.
• Models must perform contextual reasoning, not just visual detection, as evidenced by YOLOv5's failure on context-dependent objects.

Evidence (verbatim from paper)

Table 1: Experimental results for the two privacy datasets using different algorithms.

Citation

@misc{jiang2024beyond,
  title={Beyond Visual Appearances: Privacy-sensitive Objects Identification via Hybrid Graph Reasoning},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2406.12736}
}

• arXiv: 2406.12736