qhjqhj00 excelchart400k-eval
Advancing Chart Question Answering with Robust Chart Component Recognition — Hanwen Zheng et al. (2024) (arXiv:2407.21038, 2024)
What this evaluates
Evaluates a model's ability to recognize and segment specific chart components (e.g., bars, lines, pie slices, legends, axis titles) within chart images using instance segmentation.
Datasets
- ExcelChart400K — total 1698970; splits: train (1581008), val (59357), test (62605)
Metrics
mAP(primary) — range: [0, 1]- Mean Average Precision across all classes and recall levels, averaging precision values to produce a single score.
Input / output format
Input: Chart image.
Output: Instance segmentation masks (polygons) and class labels for each detected chart component.
Scoring recipe
def compute_map(predictions, ground_truth, iou_thresh=0.5):
# predictions and ground_truth are lists of dicts with 'mask', 'category', 'score'
# Calculate IoU between predicted and ground truth masks
# Match predictions to ground truth based on IoU >= threshold and highest score
# Compute precision-recall curve per class
# Average precision per class, then average across classes
pass
Common pitfalls
- Confusing instance segmentation masks with bounding boxes; mAP requires precise polygon overlap rather than box overlap.
- Failing to account for the 7 specific chart component categories, which may lead to incorrect class mapping or missed detections for titles and legends.
Evidence (verbatim from paper)
We evaluate the performance of chart component recognition with three metrics: (1) mAP (mean Average Precision) is a common metric for evaluating object detection and instance segmentation tasks. It calculates the average precision for each class across all recall levels and then averages these values to get a single score.
Citation
@misc{zheng2024advancing,
title={Advancing Chart Question Answering with Robust Chart Component Recognition},
author={Hanwen Zheng et al. (2024)},
year={2024},
note={arXiv:2407.21038}
}
- arXiv: 2407.21038