Connect The Dots Eval

connect-the-dots-eval

SketchVLM: Vision language models can annotate images to explain thoughts and guide users — Collins et al. (2026) (arXiv:2604.22875, 2026)

What this evaluates

Evaluates a model's ability to locate and connect dots in sequential order across various visual patterns. It probes precise spatial reasoning and the capacity to generate non-destructive SVG overlays that explain the reasoning process.

Datasets

• Connect-the-Dots — total 100; splits: test (100)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Binary score: 1 if the model's final text answer matches the ground truth, 0 otherwise.
• Annotation-text alignment — range: [0, 1]
  • Binary score: 1 if a VLM judge inferring the answer solely from the SVG annotations matches the model's final text answer, 0 otherwise.
• Annotation quality — range: [1, 5]
  • Rubric score from 1 to 5 evaluating the plausibility and visual clarity of the generated SVG annotations.

Input / output format

Input: Input image, task-specific prompt, optional coordinate grid, and (in multi-turn) previous annotations rendered on the image and provided as text.

Output: SVG annotations overlaid on the input image, followed by a final text answer.

Scoring recipe

def evaluate(predictions, gold):
    acc = 1.0 if predictions.text == gold.text else 0.0
    judge_ans = vlm_judge_infer(predictions.svg)
    align = 1.0 if judge_ans == predictions.text else 0.0
    qual = vlm_judge_rubric(predictions.svg, task_type) # 1-5
    return {'accuracy': acc, 'alignment': align, 'quality': qual}

Common pitfalls

• Relying solely on text accuracy while ignoring whether the SVG annotations actually support or contradict the answer.
• Coordinate system confusion: models may output coordinates on a normalized 1000x1000 scale instead of the provided grid, causing misalignment.
• VLM judge bias: alignment and quality metrics depend on a VLM-as-a-Judge, which may have systematic biases or struggle with complex SVG structures.

Evidence (verbatim from paper)

We therefore measure three distinct aspects of model performance: Accuracy serves as the primary measure of task performance and tests the effect of sketching on answer correctness. Annotation–text alignment measures how faithful the visual traces are to the text answer. We ask a VLM judge to infer the answer from the annotations alone and report how often the judge’s inferred answer matches the model’s final text answer. Annotation quality allows us to distinguish models that produce informative annotations from those that output low-quality or incoherent drawings. We adopt a VLM-as-a-Judge approach using a rubric scored from 1 to 5

Citation

@misc{collins2026sketchvlm,
  title={SketchVLM: Vision language models can annotate images to explain thoughts and guide users},
  author={Collins et al. (2026)},
  year={2026},
  note={arXiv:2604.22875}
}

• arXiv: 2604.22875