Onechart Se Eval

onechart-se-eval

OneChart: Purify the Chart Structural Extraction via One Auxiliary Token — Chen et al. (2024) (arXiv:2404.09987, 2024)

What this evaluates

Evaluates a model's ability to extract structured information from chart images, including textual OCR accuracy and precise numerical value parsing. It tests the model's capacity to convert visual chart elements into a standardized Python-dict representation, handling both annotated and unannotated charts across multiple languages and rendering styles.

Datasets

• ChartQA-SE — total 1509; splits: test (1509)
• PlotQA-SE — total 33657; splits: test (33657)
• ChartX-SE — total 2360; splits: test (2360)
• ChartY-en — total 4000; splits: test (4000)
• ChartY-zh — total 1991; splits: test (1991)

Metrics

• SCRM AP (strict/slight/high) (primary) — range: [0, 1]
  • Mean Average Precision (AP) computed on concatenated key-value tuples from the 'value' field. Three tolerance levels are used: strict (edit distance threshold = 0, relative error = 0), slight (edit distance = 2, relative error = 0.05), and high (edit distance = 5, relative error = 0.1).
• Reverse Edit distance (RE) — range: [0, 1]
  • 1 minus the normalized edit distance between the predicted and ground truth text for fields like 'title', 'source', 'x_axis', and 'y_axis'. Larger values indicate better OCR accuracy.
• ChartQA QA accuracy — range: percent
  • Standard exact-match or token-level accuracy from the ChartQA benchmark, used to evaluate downstream question-answering performance after structural extraction.

Input / output format

Input: Chart image

Output: Python-dict with keys: 'title', 'source', 'x_axis', 'y_axis', and 'value' (where 'value' is a nested dict mapping entity names to numerical data).

Scoring recipe

def compute_re(pred_text, gt_text):
    return 1 - normalized_edit_distance(pred_text, gt_text)

def compute_scrm_ap(pred_dict, gt_dict, tol='strict'):
    pred_tuples = list(pred_dict['value'].items())
    gt_tuples = list(gt_dict['value'].items())
    if tol == 'strict': ed_thr, re_thr = 0, 0
    elif tol == 'slight': ed_thr, re_thr = 2, 0.05
    else: ed_thr, re_thr = 5, 0.1
    return mean_average_precision(pred_tuples, gt_tuples, ed_thr, re_thr)

Common pitfalls

• Confusing the three SCRM tolerance levels (strict, slight, high) which use different edit distance and relative error thresholds.
• Assuming numerical annotations are always present on charts; models struggle significantly when values must be derived from axes without direct labels.
• The output must strictly conform to the specified Python-dict structure; any deviation breaks the SCRM tuple-based evaluation.

Evidence (verbatim from paper)

For the chart's textual elements such as "title", "source", "x_axis", "y_axis" in dictionary, we employ an accuracy evaluation based on normalized edit distance [39, 40]... we report the value of 1 minus the normalized editing distance as the OCR accuracy, denoted as Reverse Edit distance (RE). To evaluate the accuracy of this crucial component, we concatenate the key and item pairs into tuples and assess them using the mean Average Precision (AP) from the SCRM (Structuring Chart-oriented Representation Metric) [23].

Citation

@misc{chen2024onechart,
  title={OneChart: Purify the Chart Structural Extraction via One Auxiliary Token},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2404.09987}
}

• arXiv: 2404.09987