Fractal Pretraining Eval

fractal-pretraining-eval

Improving Fractal Pre-training — Anderson et al. (2021) (arXiv:2110.03091, 2021)

What this evaluates

Evaluates the downstream transfer capability of fractal-based pre-trained visual representations on fine-grained image classification and medical image segmentation tasks. It measures how effectively synthetic Iterated Function System (IFS) pre-training captures transferable features compared to training from scratch or using ImageNet/FractalDB pre-training.

Datasets

• CUB-2011 — total ?; splits: test (-1)
• Stanford Cars — total ?; splits: test (-1)
• Stanford Dogs — total ?; splits: test (-1)
• FGVC Aircraft — total ?; splits: test (-1)
• CIFAR-100 — total ?; splits: test (-1)
• GlaS — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Top-1 classification accuracy, calculated as the fraction of correctly predicted class labels over the total number of test instances. For segmentation tasks, performance is reported qualitatively alongside classification metrics.

Input / output format

Input: 224×224 RGB images for classification tasks; corresponding pixel-wise segmentation masks for the GlaS dataset.

Output: Discrete class labels for classification tasks; binary or multi-class segmentation masks for the GlaS dataset.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    if len(predictions) != len(gold_labels):
        raise ValueError('Prediction and gold label lengths must match')
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• The conference version of the paper contained a code bug claiming 50,000 IFS codes were used when only 1,000 were actually sampled, invalidating specific ablation results about system count vs. parameter augmentation.
• Fine-tuning hyperparameters differ significantly between tasks (150 epochs/batch 96 for classification vs. 90 epochs/batch 8 for segmentation), so cross-task performance comparisons require strict adherence to these settings.
• Evaluating synthetic fractal pre-training on natural image benchmarks (e.g., CIFAR-100, CUB) introduces a domain gap that may not reflect performance on out-of-distribution or medical imaging data.

Evidence (verbatim from paper)

We evaluate the effectiveness of the pre-trained representations by fine-tuning on several different tasks. For image classification, we use CUB-2011, Stanford Cars, Stanford Dogs, FGVC Aircraft and CIFAR-100. We also fine-tune models for medical image segmentation on the GlaS dataset. ... In fact, multi-instance prediction models can provide more than 90% of the accuracy achieved by ImageNet pre-training—and in some cases, such as for Stanford Cars, the model obtains over 98% of the ImageNet performance.

Citation

@misc{anderson2021improvingfractal,
  title={Improving Fractal Pre-training},
  author={Anderson et al. (2021)},
  year={2021},
  note={arXiv:2110.03091}
}

• arXiv: 2110.03091