Error Detection Hmc Eval

error-detection-hmc-eval

Error Detection and Constraint Recovery in Hierarchical Multi-Label Classification without Prior Knowledge — Kricheli et al. (2024) (arXiv:2407.15192, 2024)

What this evaluates

Evaluates a model's ability to detect classification errors and recover hierarchical multi-label constraints without prior knowledge. It probes the system's capacity to generate interpretable logical rules from failure patterns and improve downstream model consistency.

Datasets

• Military Vehicles — total 9444; splits: train (7555), test (1889); repo https://github.com/lab-v2/PyEDCR
• ImageNet50 — total 67500; splits: train (65000), test (2500)
• OpenImage36 — total 86400; splits: train (72000), test (14400)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. For error detection, 'total error' is defined as applying a logical OR on all per-class error predictions before computing the metric.
• balanced accuracy — range: [0, 1]
  • Average of recall obtained on each class, computed on the total error predictions (logical OR of per-class errors).
• F1-score (constraint recovery) — range: [0, 1]
  • Harmonic mean of precision and recall comparing recovered constraints against a ground truth set of constraints not used during training.

Input / output format

Input: Image inputs processed by a base vision model (ViT b_16 or DINOv2 s_14), along with model predictions, complementary granularity class outputs, and binary classification model outputs used to derive rule conditions.

Output: Binary error predictions per class and per sample, logical rules defining hierarchical constraints, and consistency scores for neurosymbolic training.

Scoring recipe

def compute_metrics(preds_per_class, gold_per_class, gold_constraints, recovered_constraints, test_set):
    # Total error detection (logical OR across per-class errors)
    total_error_pred = logical_or(preds_per_class)
    total_error_gold = logical_or(gold_per_class)
    bal_acc = balanced_accuracy(total_error_gold, total_error_pred)
    f1_err = f1_score(total_error_gold, total_error_pred)
    
    # Constraint recovery
    f1_const = f1_score(gold_constraints, recovered_constraints)
    
    # Consistency
    violations = count_samples_violating_gold_constraints(test_set)
    consistency = 1.0 - (violations / len(test_set))
    
    return bal_acc, f1_err, f1_const, consistency

Common pitfalls

• Total error is explicitly defined as the logical OR across all per-class error predictions, not a per-sample average or independent binary classification.
• Constraint recovery experiments deliberately omit labels of specific fine-grain classes during training to test noise tolerance, meaning the model must generalize without seeing those ground-truth labels.
• Rules for both f-EDR and DetRuleLearn use identical conditions; performance differences stem from the objective function optimization, not feature extraction or condition selection.

Evidence (verbatim from paper)

Results of this experiment are shown in Table[3] where we provide the balanced accuracy and F1-score of the total error - which is defined as applying a logical OR on all the per-class error classes.

Citation

@misc{kricheli2024error,
  title={Error Detection and Constraint Recovery in Hierarchical Multi-Label Classification without Prior Knowledge},
  author={Kricheli et al. (2024)},
  year={2024},
  note={arXiv:2407.15192}
}

• arXiv: 2407.15192