Deep Bias Eval

deep-bias-eval

Deep-BIAS: Detecting Structural Bias using Explainable AI — van Stein et al. (2023) (arXiv:2304.01869, 2023)

What this evaluates

Evaluates the ability of a deep learning model to detect and classify structural bias in heuristic optimization algorithms by analyzing raw performance distributions against a uniform null hypothesis.

Datasets

• BIAS toolbox heuristic pool on $f_0$ — total 336; splits: test (336)

Metrics

• detection accuracy (primary) — range: [0, 1]
  • Proportion of algorithms correctly classified as biased or non-biased compared to the baseline statistical toolbox or ground truth.
• classification precision — range: [0, 1]
  • Precision in predicting the specific type of structural bias (e.g., discretisation, clusters) among algorithms flagged as biased.

Input / output format

Input: Raw performance distributions (point positions in the search space) from executing heuristic optimizers on test function $f_0$ at dimensionality $n=30$.

Output: Class probabilities for each bias type (e.g., biased, non-biased, discretisation, clusters) and a final biased/non-biased decision.

Scoring recipe

correct_biased = sum(1 for p, g in zip(predictions, ground_truth) if p == g)
accuracy = correct_biased / len(predictions)
correct_types = sum(1 for p, g in zip(type_preds, type_truth) if p == g and g != 'non-biased')
precision = correct_types / sum(1 for g in type_truth if g != 'non-biased')

Common pitfalls

• The original statistical toolbox's Random Forest model forces class probabilities to sum to 1, whereas Deep-BIAS reserves probability mass for the uniform class, making direct probability comparison misleading.
• Boundary correction methods like 'saturate' force points to exact bounds, which statistical tests easily flag as non-uniform, but Deep-BIAS may overlook due to lacking explicit boundary context.
• Minor distributional deviations may trigger Deep-BIAS probability shifts but fail statistical p-value corrections, leading to false disagreements on bias presence.

Evidence (verbatim from paper)

It replaces 39 statistical tests with a convolutional neural network that directly predicts SB presence and type, outperforming traditional methods in both detection accuracy and classification precision.

Citation

@misc{vanstein2023deepbias,
  title={Deep-BIAS: Detecting Structural Bias using Explainable AI},
  author={van Stein et al. (2023)},
  year={2023},
  note={arXiv:2304.01869}
}

• arXiv: 2304.01869