Bongard Rwr Plus Eval

bongard-rwr-plus-eval

Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems — Pawlonka et al. (2025) (arXiv:2508.12026, 2025)

What this evaluates

Evaluates vision-language models' ability to perform abstract visual reasoning (AVR) by recognizing fine-grained, abstract visual concepts in Bongard-style matrix problems. It probes capabilities in concept selection, image-to-side classification, and free-form concept description generation.

Datasets

• Bongard-RWR+ — total 5400; splits: test (5400); repo https://github.com/pavonism/bongard-rwr-plus

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total instances. Calculated per task (CS, I1S, I2S, D1S, D2S) and reported separately or averaged.
• BLEU — range: [0, 1]
  • N-gram overlap between generated concept descriptions and reference texts.
• METEOR — range: [0, 1]
  • Harmonic mean of unigram precision and recall with synonym and stem matching.
• ROUGE_L — range: [0, 1]
  • Longest common subsequence overlap between generated and reference descriptions.
• CIDEr — range: [0, 1]
  • TF-IDF weighted n-gram similarity between generated and reference descriptions.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and reference tokens.

Input / output format

Input: For classification tasks (CS, I1S, I2S, D1S, D2S): Bongard matrix images (two sides with examples) and one or two test images, or their corresponding text captions. For concept generation (CG): raw matrix images or captions. All inputs are processed with a fixed decoding temperature of 0.5.

Output: Structured JSON output enforced via the Outlines decoding backend. Tasks require selecting a concept from a candidate set, assigning test images to matrix sides, or generating free-form text descriptions of the underlying concept.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• In I2S and D2S tasks, the prompt specifies test inputs belong to different classes, but this constraint is not enforced in the JSON output format, causing models to incorrectly assign both inputs to the same side.
• Color often acts as a visual distractor since underlying concepts are derived from structural properties of black-and-white synthetic Bongard Problems.
• Models frequently rely on high-level heuristics rather than robustly identifying subtle visual cues like contour, rotation, or precise spatial relationships.

Evidence (verbatim from paper)

We evaluate 4 state-of-the-art open-access VLMs on Bongard-RWR+... All models are evaluated at a fixed decoding temperature of 0.5, using structured JSON output format enforced via the Outlines decoding backend... Models were prompted to describe the concept behind each matrix, using either raw image inputs or InternVL2.5’s image captions. The generated responses were evaluated using standard NLP metrics: BLEU, METEOR, ROUGE_L, CIDEr, and BERTScore.

Citation

@misc{pawlonka2025bongadrwrplus,
  title={Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems},
  author={Pawlonka et al. (2025)},
  year={2025},
  note={arXiv:2508.12026}
}

• arXiv: 2508.12026