Crl Biometric Eval

crl-biometric-eval

Continual Representation Learning for Biometric Identification — Zhao et al. (2020) (arXiv:2006.04455, 2020)

What this evaluates

Evaluates a model's ability to learn generalizable biometric feature representations in a continual learning setting, specifically measuring generalization to unseen identities across sequential learning steps rather than retaining knowledge of previously seen classes.

Datasets

• CRL-face — total ?; splits: train (-1), test (-1); repo https://github.com/PatrickZH/Continual-Representation-Learning-for-Biometric-Identification
• CRL-person — total ?; splits: train (-1), test (-1); repo https://github.com/PatrickZH/Continual-Representation-Learning-for-Biometric-Identification
• LFW — total ?; splits: test (-1)
• Megaface — total ?; splits: test (-1)

Metrics

• Top 1 accuracy (primary) — range: percent
  • Percentage of correctly identified identities or matched query-gallery pairs out of the total test set, computed at the final learning step.
• mAP — range: percent
  • Mean Average Precision, calculated by averaging the Average Precision scores across all query-gallery matching pairs in the person re-identification test set.

Input / output format

Input: RGB images aligned and resized to 112x112 for face recognition, or 256x128 for person re-identification.

Output: Feature embeddings (256-dimensional for face, 2048-dimensional for person re-id). Classification/retrieval is performed via Euclidean distance similarity.

Scoring recipe

def compute_metrics(embeddings, labels, query_ids, gallery_ids):
    correct = 0
    for emb, label in zip(embeddings, labels):
        pred = argmin([euclidean_dist(emb, class_center) for class_center in class_centers])
        if pred == label: correct += 1
    top1_acc = (correct / len(labels)) * 100
    
    dists = pairwise_euclidean(query_embs, gallery_embs)
    mAP = mean([ap(dists[i], query_labels[i], gallery_labels) for i in range(len(query_embs))])
    return top1_acc, mAP

Common pitfalls

• Evaluating performance on old (seen) classes instead of unseen classes, as the paper shows old-class performance does not decrease significantly and is unsuitable for measuring CRL.
• Ignoring the continual learning setting (5-step or 10-step sequential training) and treating it as standard single-step training.
• Not reporting standard deviation over 5 random seeds/runs, as the protocol requires repeating experiments five times.

Evidence (verbatim from paper)

We evaluate model’s generalization ability on unseen classes in CRL setting, which is more suitable. ... Top 1 accuracy (%) is reported. ... Top1 and mAP accuracy (%) in the final learning step.

Citation

@misc{zhao2020continual,
  title={Continual Representation Learning for Biometric Identification},
  author={Zhao et al. (2020)},
  year={2020},
  note={arXiv:2006.04455}
}

• arXiv: 2006.04455