qhjqhj00 ccc-eval
International Workshop on Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines — Shahbaz et al. (2021) (arXiv:2110.14613, 2021)
What this evaluates
Evaluates continual semi-supervised learning on crowd counting by measuring how well a model adapts to evolving unlabeled data streams across sequential sessions.
Datasets
- Continual Crowd Counting (CCC) — total ?; splits: train (-1), val (-1), test (-1)
Metrics
Mean Absolute Error (MAE)(primary) — range: [0, inf)- Average absolute difference between predicted and true crowd counts across all test instances: MAE = (1/N) * sum(|y_pred - y_true|).
Input / output format
Input: Sequential crowd counting image streams divided into sessions, containing unlabeled validation and test folds for incremental self-training.
Output: Predicted crowd count/density for each image in the stream.
Scoring recipe
def compute_mae(predictions, gold):
if len(predictions) != len(gold):
raise ValueError('Length mismatch')
errors = [abs(p - g) for p, g in zip(predictions, gold)]
return sum(errors) / len(errors)
Common pitfalls
- Comparing MAE values directly to standard batch-trained crowd counting papers without noting the different training protocol (400 vs 800 supervised images).
- Treating validation and test folds as a single contiguous stream when the protocol specifies separate incremental updates.
- Assuming incremental updates always improve performance; the paper notes updates are not always effective on validation streams.
Evidence (verbatim from paper)
Table 2 shows a quantitative analysis of the performance of the fine-tuned supervised model (sup) and two incrementally updated models (upd) on the validation and the test split, respectively, using the mean absolute error (MAE) metric.
Citation
@misc{shahbaz2021cssl,
title={International Workshop on Continual Semi-Supervised Learning: Introduction, Benchmarks and Baselines},
author={Shahbaz et al. (2021)},
year={2021},
note={arXiv:2110.14613}
}
- arXiv: 2110.14613