qhjqhj00 cld-extraction-eval
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion — Leitch (2026) (arXiv:2604.18566, 2026)
What this evaluates
Evaluates LLMs' ability to extract structured Causal Loop Diagrams (CLDs) from natural language system dynamics descriptions. It probes structured output generation, schema conformance, and iterative model updating under varying context lengths and prompt strategies.
Datasets
- CLD Leaderboard — total 53; splits: test (53); repo https://github.com/tleitch/sd-ai
Metrics
exact_structured_match(primary) — range: [0, 1]- Pass rate based on exact structured match against a ground-truth schema. A response scores 1 if the extracted JSON matches all required fields (variable names, edge directions, polarities, cardinality constraints) after normalizing variable name casing and whitespace; otherwise 0. Reasoning traces are ignored.
Input / output format
Input: Natural language text passage describing a system dynamics model, optionally containing an existing CLD for iterative updates.
Output: A JSON object encoding variables, directed links, polarities, and optionally loop labels. Only the final JSON is evaluated; reasoning traces are ignored.
Scoring recipe
def score(prediction, gold):
pred_json = extract_final_json(prediction)
gold_json = gold
if normalize_casing_whitespace(pred_json) == normalize_casing_whitespace(gold_json):
return 1.0
return 0.0
# Average across all 53 tests
Common pitfalls
- Reasoning traces or chain-of-thought content prior to the final JSON are explicitly ignored in scoring; only the final structured output is evaluated.
- Variable name casing and whitespace must be normalized before comparison; raw string matching will fail.
- Models must not hallucinate extra variables or omit required links to pass.
Evidence (verbatim from paper)
Tests are evaluated by exact structured match against a ground-truth schema: a response is scored pass if and only if the extracted JSON matches the ground-truth on all required fields (variable names, edge directions, polarities, and any specified cardinality constraints), after normalisation of variable name casing and whitespace. Reasoning traces or chain-of-thought content prior to the final JSON are ignored in scoring; only the final structured output is evaluated.
Citation
@misc{leitch2026benchmarking,
title={Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion},
author={Leitch (2026)},
year={2026},
note={arXiv:2604.18566}
}
- arXiv: 2604.18566