Dcg Bench Eval

dcg-bench-eval

OpusAnimation: Code-Based Dynamic Chart Generation — Bozheng Li et al. (2025) (arXiv:2510.03341, 2025)

What this evaluates

Evaluates multimodal large language models' ability to generate executable HTML/JavaScript code for dynamic chart animations from text or video prompts. It probes instruction following, code executability, and fine-grained semantic alignment between generated visualizations and input specifications.

Datasets

• DCG-8K — total 8000; splits: test (-1)

Metrics

• Execution Pass Rate (primary) — range: percent
  • Binary indicator (1 if generated code renders a valid, non-blank animation video, 0 otherwise). Overall rate is the proportion of successful executions across test samples.
• QA-based Score — range: [0, 1]
  • Average of binary answers from an MLLM evaluator (Gemini-2.5-Pro) to ~10 targeted QA pairs per modality. Formulas: S_code = (1/N_c) * sum(MLLM_eval(c_g, QA_code_i)) and S_video = (1/N_v) * sum(MLLM_eval(v_g, QA_video_j)). Returns 0 if code fails to render.

Input / output format

Input: A query q containing a data sequence d and a modality-specific description: either a detailed text prompt (t_d), a simple text prompt (t_s), or a reference animation video (v).

Output: Complete HTML/JavaScript code snippet c_g that renders a dynamic chart animation video v_g.

Scoring recipe

def evaluate(code_output, qa_code, qa_video, evaluator):
    # 1. Execution Pass Rate
    try:
        video = render_html_to_video(code_output)
        is_valid = video is not None and not is_blank(video)
    except Exception:
        is_valid = False
    pass_rate = 1.0 if is_valid else 0.0

    # 2. QA-based Scores
    if not is_valid:
        return pass_rate, 0.0, 0.0

    code_score = sum(1.0 if evaluator(code_output, qa) else 0.0 for qa in qa_code) / len(qa_code)
    video_score = sum(1.0 if evaluator(video, qa) else 0.0 for qa in qa_video) / len(qa_video)
    return pass_rate, code_score, video_score

Common pitfalls

• The QA-based scores default to 0 if the generated code fails to render, so syntactically valid but non-executable code is heavily penalized.
• Evaluation relies on a proprietary MLLM (Gemini-2.5-Pro) for semantic scoring, making exact reproduction of QA-based scores difficult without identical evaluator versions and prompts.
• The exact size of the test split (DCG-Bench) is not explicitly stated, requiring careful handling of split boundaries when re-running.

Evidence (verbatim from paper)

Execution Pass Rate. We first assess basic executability by verifying whether generated code $c_{g}$ renders a valid, non-blank animation video $v_{g}$. Each instance then receives a binary indicator, and overall pass rate is computed as proportion of successful executions across the test samples. QA-based Scores. To better assess the semantic alignment between generated outputs and the input specifications in dynamic chart generation, we propose a QA-based metric using powerful MLLM Team et al. ([2023]). For each chart, we construct around ten targeted QA pairs based on its reference code or video, focusing on fine-grained aspects such as animation order, element appearance, and timing consistency... Based on these QA sets, we define the evaluation scores $S_{\text{code}}$ and $S_{\text{video}}$ as: $S_{code}(q,c_{g}) = \frac{1}{N_{c}}\sum^{N_{c}}{i=1}\text{MLLM}{eval}(c_{g},QA^{(i)}{code})$ ... Here, $\text{MLLM}{eval}$ employs Gemini2.5-Pro Team et al. ([2023]) and returns 1 if the artifact meets the corresponding QA requirement, and 0 otherwise.

Citation

@misc{li2025opusanimation,
  title={OpusAnimation: Code-Based Dynamic Chart Generation},
  author={Bozheng Li et al. (2025)},
  year={2025},
  note={arXiv:2510.03341}
}

• arXiv: 2510.03341