Open Set Recognition Eval

open-set-recognition-eval

Learning a Neural-network-based Representation for Open Set Recognition — Hassen et al. (2018) (arXiv:1802.04365, 2018)

What this evaluates

This evaluation probes a model's ability to correctly classify instances into known classes while accurately detecting and rejecting instances from unknown, unseen classes. It measures both outlier detection capability via ROC analysis and multi-class recognition performance including an explicit unknown category.

Datasets

• MNIST — total ?; splits: train (-1), val (-1), test (-1)
• MS Challenge — total 10260; splits: train (-1), val (-1), test (-1)
• Android Genome — total 986; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, computed using the model's outlier score to calculate True Positive Rate and False Positive Rate at varying thresholds. Reported up to 100% FPR and 10% FPR.
• F-Score — range: [0, 1]
  • Harmonic mean of precision and recall. Calculated per class (K known classes + 1 unknown class) and then macro-averaged across all K+1 classes.

Input / output format

Input: Raw images (MNIST) or extracted function call graph features represented as adjacency matrices/edge frequency vectors (malware datasets).

Output: A continuous outlier score for detection tasks, or a discrete class label (one of K known classes or unknown) for recognition tasks, determined by comparing the outlier score to a threshold.

Scoring recipe

# Outlier Detection (AUC)
scores = model.compute_outlier_score(X_test)
y_true = [1 if x in unknown_classes else 0 for x in X_test]
auc = roc_auc_score(y_true, scores) # computed up to 100% or 10% FPR

# Open Set Recognition (F-Score)
preds = []
for x, score in zip(X_test, scores):
    if score > threshold:
        preds.append("unknown")
    else:
        preds.append(model.predict_class(x))
f1s = []
for label in known_classes + ["unknown"]:
    p, r, f1, _ = precision_recall_fscore_support(y_true, preds, labels=[label])
    f1s.append(f1)
avg_f1 = sum(f1s) / len(f1s)

Common pitfalls

• The threshold for classifying an instance as unknown must be tuned using only known-class validation data, not the full test set.
• AUC should be reported at restricted FPR ranges (e.g., ≤10%) for practical relevance, as 100% FPR AUC can mask poor performance at low false positive rates.
• Malware datasets require specific feature extraction (FCG adjacency matrices) and class filtering (≥40 samples) to ensure valid train/val/test splits.

Evidence (verbatim from paper)

We use average precision, recall and f-score metrics to evaluate open set recognition performance and t-test for statistical significance. Precision, recall and f-score are first calculated for each of the K known class labels and the one “unknown” label. Then the average overall the K+1 classes is calculated.

Citation

@misc{hassen2018learning,
  title={Learning a Neural-network-based Representation for Open Set Recognition},
  author={Hassen et al. (2018)},
  year={2018},
  note={arXiv:1802.04365}
}

• arXiv: 1802.04365