Eee Bench Eval

eee-bench-eval

EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark — Li et al. (2024) (arXiv:2411.01492, 2024)

What this evaluates

Probes multimodal reasoning and visual diagram interpretation in electrical and electronics engineering. It tests whether models can integrate complex circuit and system diagrams with textual problem descriptions to apply domain-specific knowledge and perform accurate calculations or logical deductions.

Datasets

• EEE-Bench — total 2860; splits: test (2860)

Metrics

• accuracy (primary) — range: percent
  • Accuracy is computed as the proportion of correctly answered questions after extracting the concise final answer from the model's raw output and comparing it to the ground truth.

Input / output format

Input: A multimodal instance consisting of an image (e.g., circuit diagram, system schematic, or technical figure) paired with a textual question or problem statement in the electrical and electronics engineering domain.

Output: A concise final answer extracted from the model's raw generated response.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        if pred.strip().lower() == g.strip().lower():
            correct += 1
    return correct / len(gold)

Common pitfalls

• Models exhibit a 'laziness' bias, often ignoring visual inputs and relying solely on text, which skews performance on diagram-heavy tasks.
• The evaluation pipeline uses GPT-4o-mini to extract concise answers from raw model outputs, which may introduce extraction inconsistencies or bias compared to direct string matching.
• Zero-shot evaluation setup may not reflect real-world engineering workflows that typically provide few-shot examples or step-by-step reasoning prompts.

Evidence (verbatim from paper)

Raw responses are first generated from LMMs and then processed by GPT-4o-mini to extract the concise answers given an exemplary system prompt. Finally, accuracy is obtained through calculation against the ground truth.

Citation

@misc{li2024eeebench,
  title={EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2411.01492}
}

• arXiv: 2411.01492