Kvg Bench Eval

kvg-bench-eval

DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding — Ma et al. (2025) (arXiv:2503.12797, 2025)

What this evaluates

Evaluates knowledge-intensive visual grounding (KVG), requiring models to combine domain-specific reasoning with fine-grained visual perception to locate specific entities in images containing multiple similar objects.

Datasets

• KVG-Bench — total 1300; splits: test (1300); repo https://github.com/thunlp/DeepPerception

Metrics

• accuracy (primary) — range: percent
  • Percentage of test cases where the predicted bounding box achieves an Intersection over Union (IoU) with the ground truth that meets or exceeds a threshold τ. Correct predictions are counted as 1, others as 0, then averaged.

Input / output format

Input: An image containing multiple similar entities, accompanied by a natural language question/prompt specifying the target entity and often requiring domain knowledge to disambiguate.

Output: A structured response containing a reasoning chain enclosed in ... tags, followed by the predicted bounding box coordinates enclosed in ... tags.

Scoring recipe

def compute_accuracy(predictions, gold_boxes, threshold=0.5):
    correct = 0
    for pred, gold in zip(predictions, gold_boxes):
        pred_box = extract_box_from_answer_tags(pred)
        iou = calculate_iou(pred_box, gold)
        if iou >= threshold:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models may output degenerate bounding boxes (e.g., [0,0,1000,1000]) to exploit weak rewards; the evaluation enforces a strict IoU threshold τ to prevent this.
• Failure to adhere to the ... and ... format tags leads to parsing errors and invalid predictions.
• Domain knowledge is required to distinguish visually similar objects; models relying solely on visual features without reasoning will fail.

Evidence (verbatim from paper)

DeepPerception introduces knowledge-intensive visual grounding (KVG), a task requiring fine-grained perception and domain-specific reasoning, to bridge the gap between MLLMs' knowledge and human-like visual cognition. ... enabling superior accuracy (+8.08%) and cross-domain generalization (+4.60%) over baseline methods on KVG-Bench, a manually curated benchmark spanning 10 domains with 1.3K test cases. The IoU reward evaluates the spatial alignment between predicted bounding boxes and ground-truth annotations. Given the ground-truth bounding box B and the predicted bounding box ilde{B}, the IoU reward is formally defined as: R_IoU = IoU(B, ilde{B}) if IoU >= τ else 0.

Citation

@misc{ma2025deepperception,
  title={DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding},
  author={Ma et al. (2025)},
  year={2025},
  note={arXiv:2503.12797}
}

• arXiv: 2503.12797