F Siol 310 Eval

f-siol-310-eval

F-SIOL-310: A Robotic Dataset and Benchmark for Few-Shot Incremental Object Learning — Ayub et al. (2021) (arXiv:2103.12242, 2021)

What this evaluates

Evaluates few-shot incremental learning (FSIL) capabilities in robotic vision, specifically testing a model's ability to learn new object classes sequentially with very limited examples (5 or 10 per class) while resisting catastrophic forgetting of previously learned classes.

Datasets

• F-SIOL-310 — total 310; splits: train (-1), test (-1)

Metrics

• classification accuracy (%) (primary) — range: percent
  • Percentage of correctly classified images out of the total evaluated images at each incremental step. Reported as mean ± standard deviation across 10 independent runs with different random seeds.
• average incremental accuracy — range: percent
  • The arithmetic mean of the classification accuracies achieved by the model after each of the 11 incremental steps (2 classes per step).

Input / output format

Input: RGB images resized to 256×256 and randomly cropped to 224×224, paired with class labels. Models process images through a backbone network (e.g., ResNet18) to extract features, followed by a classifier.

Output: Class label prediction for each input image.

Scoring recipe

accuracies = []
for increment in 1..11:
    predictions = model.evaluate(test_set_increment)
    acc = (predictions == gold_labels).mean() * 100
    accuracies.append(acc)
avg_inc_acc = sum(accuracies) / len(accuracies)
report mean and std of avg_inc_acc over 10 random seeds

Common pitfalls

• Models are highly sensitive to the order of class increments, leading to significantly high variance across random seeds.
• Increasing shots from 5 to 10 rarely improves accuracy for most methods, as they still require more data per class to generalize effectively.
• Performance heavily depends on the quality of the pre-trained backbone features; weaker base features significantly degrade accuracy for fixed-feature methods.

Evidence (verbatim from paper)

For evaluation, we tested each approach in each increment on all the classes it had learned so far, resulting in decreasing accuracy curves. We also report average incremental accuracy which is the average of all the accuracies achieved by a model for all the increments. For robustness, we ran all the experiments for all the models 10 times with different random seeds and report average and standard deviation of the accuracies.

Citation

@misc{ayub2021fsiol310,
  title={F-SIOL-310: A Robotic Dataset and Benchmark for Few-Shot Incremental Object Learning},
  author={Ayub et al. (2021)},
  year={2021},
  note={arXiv:2103.12242}
}

• arXiv: 2103.12242