Scratheval Eval

scratheval-eval

ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming — Si et al. (2026) (arXiv:2602.00757, 2026)

What this evaluates

Evaluates LLMs' ability to understand, diagnose, and repair bugs in multimodal, event-driven block-based programming environments (Scratch). It probes functional correctness, structured bug explanation, trigger/mechanism identification, and patch minimality/semantic preservation.

Datasets

• ScratchEval — total 100; splits: test (100)

Metrics

• G-Acc (primary) — range: [0, 1]
  • Accuracy of recovering the project-level intent and functional structure (e.g., project goal, main loop, core sprites).
• U-Acc — range: [0, 1]
  • Strict joint accuracy where the model's structured explanation correctly identifies the trigger, mechanism, and outcome matching the reference.
• T-F1 — range: [0, 1]
  • F1-score for identifying the specific event or action that initiates faulty behavior against gold-standard triggers.
• M-Acc — range: [0, 1]
  • Accuracy of classifying the underlying causal bug mechanism (e.g., concurrency/timing, state initialization, missing handlers).
• Fix Success Rate — range: [0, 1]
  • Percentage of bugs where the model's generated patch passes all executable VM-level test suites.

Input / output format

Input: Buggy Scratch project exported as .sb3 (containing blocks, sprites, and metadata) paired with original multimedia assets (PNG/SVG images, WAV sounds). No gameplay videos are provided.

Output: Structured bug explanation (trigger, mechanism, outcome) and/or a repaired project patch (modified blocks/sprites). For repair tasks, the output is evaluated as a complete .sb3 project.

Scoring recipe

def score_scratheval(predictions, gold):
    g_acc = 1.0 if predictions.project_intent == gold.project_intent else 0.0
    u_acc = 1.0 if (predictions.trigger == gold.trigger and
                    predictions.mechanism == gold.mechanism and
                    predictions.outcome == gold.outcome) else 0.0
    t_f1 = compute_f1(gold.triggers, predictions.triggers)
    m_acc = 1.0 if predictions.mechanism_tag == gold.mechanism_tag else 0.0
    fix_success = 1.0 if execute_vm_tests(predictions.patch) == 'PASS' else 0.0
    return {'G-Acc': g_acc, 'U-Acc': u_acc, 'T-F1': t_f1, 'M-Acc': m_acc, 'Fix Success': fix_success}

Common pitfalls

• Models frequently over-edit or introduce auxiliary state/logic beyond the localized fix, violating minimality.
• Patches often exhibit semantic drift, passing tests but altering unintended project behaviors compared to the original intent.
• Models apply sequential-code heuristics that fail to respect Scratch’s event-driven concurrency and broadcast ordering.

Evidence (verbatim from paper)

Global Understanding (G-Acc) = accuracy of recovering the project-level intent/structure; Bug Understanding (U-Acc) = percentage of bugs where the model’s structured explanation matches the reference (trigger, mechanism, and outcome); T-F1 = trigger identification F1-score; M-Acc = mechanism tag accuracy; Fix Gains = number of previously failing repairs that became correct after LoRA tuning (i.e., fail→success relative to Qwen zero-shot on the 100-bug evaluation set).

Citation

@misc{si2026scratheval,
  title={ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming},
  author={Si et al. (2026)},
  year={2026},
  note={arXiv:2602.00757}
}

• arXiv: 2602.00757