C2f Chart Eval

c2f-chart-eval

C2F-CHART: A Curriculum Learning Approach to Chart Classification — Shaheen et al. (2024) (arXiv:2409.04683, 2024)

What this evaluates

Evaluates a model's ability to classify chart types from images using a coarse-to-fine curriculum learning approach. It measures performance on broad and fine-grained chart categories to assess hierarchical classification capabilities.

Datasets

• ICPR 2022 UB Unitec PMC Dataset — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1-score (primary) — range: percent
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Reported as a percentage.
• Precision — range: percent
  • Ratio of correctly predicted positive observations to the total predicted positives.
• Recall — range: percent
  • Ratio of correctly predicted positive observations to all observations in the actual class.

Input / output format

Input: RGB image of a chart.

Output: Predicted chart type class label.

Scoring recipe

def compute_macro_f1(preds, gold):
    classes = set(gold) | set(preds)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return sum(f1_scores) / len(f1_scores) * 100

Common pitfalls

• The best-performing level-1 checkpoint does not guarantee the best level-2 performance; a combinatorial search over multiple checkpoints is required.
• Model ensembling (averaging logits) outperforms model souping (averaging weights) for this task, contrary to some prior assumptions.

Evidence (verbatim from paper)

We benchmarked our results on the testing dataset, called ICPR 2022 UB Unitec PMC Dataset, and compared them with previous work. ... our testing precision, recall, and F1-score demonstrate superior performance to all competition participants and Swin-Chart.

Citation

@misc{shaheen2024c2fchart,
  title={C2F-CHART: A Curriculum Learning Approach to Chart Classification},
  author={Shaheen et al. (2024)},
  year={2024},
  note={arXiv:2409.04683}
}

• arXiv: 2409.04683