Pbench Eval

pbench-eval

Falcon Perception — Bevli et al. (2026) (arXiv:2603.27365, 2026)

What this evaluates

Evaluates a model's ability to perform referring expression segmentation across five hierarchical levels of semantic complexity, from basic object recognition to fine-grained attribute binding, OCR-based disambiguation, spatial layout understanding, and relational interactions. It also stress-tests long-context generation and instance stability in crowded scenes with high object counts.

Datasets

• PBench — total 5400; splits: test (5400); repo https://github.com/tiiuae/Falcon-Perception

Metrics

• per-level performance (primary) — range: [0, 1]
  • Compute Intersection over Union (IoU) between the predicted mask and ground-truth mask. A prediction is correct if IoU ≥ 0.5. Accuracy is calculated per complexity level (0–4) and as an overall average to produce a capability profile.

Input / output format

Input: A single image paired with a natural language referring expression prompt (e.g., 'red car', 'Diet Coke bottle', 'person holding umbrella').

Output: A sequence of structured predictions per instance: coordinates, size, and segmentation mask, generated autoregressively via a chain-of-perception decoding interface.

Scoring recipe

def score(predictions, gold_masks, levels, threshold=0.5):
    correct = {l: 0 for l in range(5)}
    total = {l: 0 for l in range(5)}
    for pred, gt, lvl in zip(predictions, gold_masks, levels):
        iou = compute_iou(pred, gt)
        if iou >= threshold:
            correct[lvl] += 1
        total[lvl] += 1
    per_level_acc = {l: correct[l]/total[l] for l in range(5)}
    overall_acc = sum(correct.values()) / sum(total.values())
    return per_level_acc, overall_acc

Common pitfalls

• Existing benchmarks conflate spatial ambiguity, complex prompts, and world knowledge into a single score, masking specific failure modes.
• Models may suffer from instance duplication, drift, or 'object not found' prior collapse when generating masks for crowded scenes (K > 150).
• Reporting only an aggregate score hides capability gaps; the benchmark requires per-level profiling to diagnose weaknesses (e.g., OCR vs. spatial grounding).

Evidence (verbatim from paper)

We therefore report per-level performance in addition to an overall average, yielding a capability profile that reveals where a model fails (e.g., collapsing at OCR or relations).

Citation

@misc{bevli2026falconperception,
  title={Falcon Perception},
  author={Bevli et al. (2026)},
  year={2026},
  note={arXiv:2603.27365}
}

• arXiv: 2603.27365