Deart Eval

deart-eval

DEArt: Dataset of European Art — Reshetnikov et al. (2022) (arXiv:2211.01226, 2022)

What this evaluates

Evaluates object detection and pose classification capabilities on historical European paintings. Probes a model's ability to recognize culturally heritage-specific entities and human-like poses in artistic contexts rather than natural photographs.

Datasets

• DEArt — total 15000; splits: train (10500), val (2250), test (2250)

Metrics

• mAP@0.5 (primary) — range: [0, 1]
  • Mean average precision at an Intersection over Union (IoU) threshold of 0.5, averaged across all 69 object classes.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for pose classification across 12 classes, reported as both macro and weighted averages.

Input / output format

Input: RGB images of European paintings (XII–XVIII century) with bounding box annotations for 69 object classes and pose labels for human-like objects.

Output: Bounding box coordinates, class labels for detected objects, and pose classification labels for human-like objects.

Scoring recipe

def compute_mAP_at_05(predictions, gold, iou_thresh=0.5):
    aps = []
    for cls in classes:
        cls_preds = [p for p in predictions if p['class'] == cls]
        cls_gold = [g for g in gold if g['class'] == cls]
        ap = average_precision(cls_preds, cls_gold, iou_thresh)
        aps.append(ap)
    return mean(aps)

def compute_f1(predictions, gold):
    return f1_score(gold, predictions, average='weighted')

Common pitfalls

• The dataset is highly unbalanced, causing minority pose classes (e.g., 'ride', 'fall') to severely drag down overall F1 scores.
• Models pretrained on natural image datasets (MS COCO, PASCAL VOC) suffer significant precision drops on artistic styles compared to domain-specific training.
• Complex or non-iconic images can act as noise and contaminate training if the model lacks capacity to capture visual variability across many classes.

Evidence (verbatim from paper)

Given that our dataset is in Pascal VOC format, we chose AP@0.5 per class and mAP@0.5 as evaluation metrics. ... Evaluation over all 12 classes shows F1=0.471, with weighted F1=0.89.

Citation

@misc{reshetnikov2022deart,
  title={DEArt: Dataset of European Art},
  author={Reshetnikov et al. (2022)},
  year={2022},
  note={arXiv:2211.01226}
}

• arXiv: 2211.01226