Circuit Eval

circuit-eval

CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs — Skelic et al. (2025) (arXiv:2502.07980, 2025)

What this evaluates

Evaluates large language models' ability to interpret analog circuit diagrams and netlists, and perform multi-level reasoning to calculate correct numerical values for circuit parameters.

Datasets

• CIRCUIT — total 510; splits: test (510)

Metrics

• accuracy (primary) — range: [0, 1]
  • A response is marked correct if the absolute difference between the model's predicted numerical answer and the ground truth is less than 0.001. The paper reports both global and template accuracy variants.

Input / output format

Input: Circuit diagrams (for a subset of questions), circuit netlists (for a subset), and natural language questions. Prompts vary by experiment (zero-shot/one-shot, with/without netlists).

Output: Final numerical answers formatted to six decimal places.

Scoring recipe

def compute_accuracy(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        if abs(float(pred) - float(gt)) < 0.001:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Models must output exactly six decimal places as instructed, but scoring only checks numerical proximity (<0.001).
• Automatic scoring checks numerical accuracy, while human evaluation checks for any errors including topology misunderstandings, formatting, or reasoning flaws.
• Netlists are only provided in specific experimental setups, so models must rely on diagram interpretation when netlists are absent.

Evidence (verbatim from paper)

We used an automatic evaluation method to assess model responses and reported both global and template accuracies. Responses were deemed correct if the absolute difference from the ground truth was less than 0.001.

Citation

@misc{skelic2025circuit,
  title={CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs},
  author={Skelic et al. (2025)},
  year={2025},
  note={arXiv:2502.07980}
}

• arXiv: 2502.07980