Fewshot Fmri Decoding Eval

fewshot-fmri-decoding-eval

Few-shot Decoding of Brain Activation Maps — Bontonou et al. (2020) (arXiv:2010.12500, 2020)

What this evaluates

Evaluates few-shot learning methods for decoding brain activation maps from fMRI data. It probes a model's ability to classify cognitive tasks using very limited labeled examples (1 or 5 per class) by randomly sampling novel classes and instances.

Datasets

• IBC dataset — total 2120; splits: train (-1), val (-1), test (-1); repo https://github.com/mbonto/fewshot_neuroimaging_classification

Metrics

• accuracy (primary) — range: [0, 1]
  • Classification accuracy averaged over 10,000 randomly synthesized few-shot tasks. Each task consists of a support set (1 or 5 shots per class) and a query set (15 samples per class) from 5 novel classes.

Input / output format

Input: fMRI brain activation maps (processed through backbones like MLP, GNN, or CNN) for a support set of labeled examples and a query set of unlabeled examples from 5 novel cognitive task classes.

Output: Class label prediction for each query sample from the 5 novel classes.

Scoring recipe

def compute_accuracy(predictions, labels, n_tasks=10000, shots=5):
    correct = 0
    total = 0
    for _ in range(n_tasks):
        support, query = sample_fewshot_task(classes, shots=shots)
        preds = model.predict(query, support)
        correct += sum(p == l for p, l in zip(preds, query.labels))
        total += len(query.labels)
    return correct / total

Common pitfalls

• Tasks are synthesized on-the-fly by randomly sampling classes and samples, so results are averaged over 10,000 random tasks rather than evaluated on a fixed test set.
• Hyperparameters are tuned on a validation split of 500 tasks, not directly on the test/novel split.
• The baseline uses raw features with Nearest Class Mean (NCM), while few-shot methods use learned backbones, making direct comparison sensitive to feature extraction quality.

Evidence (verbatim from paper)

Following the trend in few-shot learning, we consider the two most common case studies: 5-way 5-shot 15-query and 5-way 1-shot 15-query tasks. In practice, other settings could be studied depending on the application at stake. The tasks are synthesized by randomly sampling classes and samples in these classes. To obtain statistically robust results on the novel dataset, we average the accuracy over 10000 tasks and we report the confidence interval at 95%.

Citation

@misc{bontonou2020fewshot,
  title={Few-shot Decoding of Brain Activation Maps},
  author={Bontonou et al. (2020)},
  year={2020},
  note={arXiv:2010.12500}
}

• arXiv: 2010.12500