Chartcap Eval

chartcap-eval

ChartCap: Mitigating Hallucination of Dense Chart Captioning — Lim et al. (2025) (arXiv:2508.03164, 2025)

What this evaluates

Evaluates the capability of vision-language models to generate dense, structurally accurate captions for charts while mitigating hallucinations. It probes both reference-based text similarity and a novel visual consistency metric that verifies if a generated caption can successfully reconstruct the original chart image.

Datasets

• ChartCap — total 565000; splits: train (509000), test (-1)

Metrics

• Visual Consistency Score (primary) — range: [0, 1]
  • Regenerates a chart image from the generated caption using an LLM-to-Python code pipeline, then computes a visual similarity score between the regenerated and original chart images.
• SacreBLEU — range: [0, 1]
  • Standard BLEU score with SacreBLEU tokenization and smoothing, measuring n-gram overlap between generated and reference captions.
• ROUGE — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation, measuring recall of overlapping n-grams or longest common subsequences between predictions and gold captions.
• BERTScore — range: [0, 1]
  • Computes cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1 score.
• OCRScore — range: [0, 1]
  • Measures alignment and fidelity between text elements in the generated caption and ground-truth chart text/OCR data.

Input / output format

Input: Chart image paired with the instruction: 'Please provide a detailed caption for the chart.'

Output: A single text string containing the dense caption for the chart.

Scoring recipe

def evaluate(caption, gold_caption, original_chart_img):
    # Reference-based metrics
    bleu = sacrebleu.corpus_bleu([caption], [[gold_caption]])
    rouge = compute_rouge(caption, gold_caption)
    bert = compute_bertscore(caption, gold_caption)
    
    # Visual Consistency Score
    code = llm_to_python_code(caption)
    regenerated_img = execute_code(code)
    vcs = image_similarity(original_chart_img, regenerated_img)
    
    # OCRScore
    ocr = text_alignment_score(caption, extract_ocr(original_chart_img))
    
    return {'SacreBLEU': bleu, 'ROUGE': rouge, 'BERTScore': bert, 'VCS': vcs, 'OCRScore': ocr}

Common pitfalls

• Reference-based metrics (BLEU/ROUGE) fail to capture factual hallucination or absolute caption quality, as noted by the authors.
• VCS depends on the LLM's ability to correctly translate text back to plotting code, which can introduce generation errors unrelated to the caption's actual quality.
• Human evaluation relies on a small sample size (100 captions) and crowd workers, which may lack domain expertise for complex chart structures.

Evidence (verbatim from paper)

For metrics, we use SacreBLEU, ROUGE, METEOR, and BERTScore, with our Visual Consistency Score and OCRScore.

Citation

@misc{lim2025chartcap,
  title={ChartCap: Mitigating Hallucination of Dense Chart Captioning},
  author={Lim et al. (2025)},
  year={2025},
  note={arXiv:2508.03164}
}

• arXiv: 2508.03164