Fractaldb Pretrain Eval

fractaldb-pretrain-eval

Pre-training without Natural Images — Kataoka et al. (2021) (arXiv:2101.08515, 2021)

What this evaluates

Evaluates the effectiveness of pre-training convolutional neural networks on automatically generated fractal image datasets (FractalDB) compared to natural image pre-training and self-supervised learning, measuring downstream classification accuracy on standard benchmarks.

Datasets

• CIFAR-10 — total ?; splits: train (-1), test (-1)
• CIFAR-100 — total ?; splits: train (-1), test (-1)
• ImageNet-100 — total ?; splits: train (-1), test (-1)
• Places-30 — total ?; splits: train (-1), test (-1)
• ImageNet-1k — total ?; splits: train (-1), test (-1)
• Places-365 — total ?; splits: train (-1), test (-1)
• Pascal VOC 2012 — total ?; splits: train (-1), test (-1)
• Omniglot — total ?; splits: train (-1), test (-1)

Metrics

• classification accuracy (primary) — range: percent
  • Percentage of correctly classified instances out of the total number of instances in the evaluation set.

Input / output format

Input: 224x224 pixel RGB images (cropped from 256x256 inputs) with class labels.

Output: Predicted class label (integer index or category name) for each image.

Scoring recipe

def compute_classification_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• The paper uses reduced-category subsets (ImageNet-100, Places-30) for hyperparameter exploration, which can overestimate performance compared to full benchmarks.
• Fine-tuning hyperparameters (batch size 256, LR 0.01 with step decay at epochs 30/60) are fixed across all datasets, deviating from standard dataset-specific fine-tuning protocols.
• Pre-training uses 'formula-supervision' (mathematically generated labels) rather than human annotations, making direct accuracy comparisons with supervised baselines sensitive to label quality and category alignment.

Evidence (verbatim from paper)

Table 6: Classification accuracies of the Ours (FractalDB-1k/10k), Scratch, DeepCluster-10k (DC-10k), ImageNet-100/1k and Places-30/365 pre-trained models on representative pre-training datasets. We show the types of pre-trained image (Pre-train Img; which includes {Natural Image (Natural), Formula-driven Image (Formula)}) and Supervision types (Type; which includes {Self-supervision, Supervision, Formula-supervision}).

Citation

@misc{kataoka2021pretraining,
  title={Pre-training without Natural Images},
  author={Kataoka et al. (2021)},
  year={2021},
  note={arXiv:2101.08515}
}

• arXiv: 2101.08515