Biasinear Eval

biasinear-eval

Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations — Wei et al. (2026) (arXiv:2602.01030, 2026)

What this evaluates

Evaluates the robustness and sensitivity of multimodal large language models (MLLMs) to perturbations in spoken multiple-choice questions. It probes how models handle variations in language, accent, speaker gender, and answer option ordering, measuring both absolute correctness and prediction stability across conditions.

Datasets

• BiasInEar — total ?; splits: test (-1); repo https://github.com/ntunlplab/BiasInEar

Metrics

• Question Entropy (primary) — range: [0, 1]
  • Shannon entropy of the model's answer distribution across four options, normalized by base 4 to ensure values fall in [0, 1]. Formula: H_q = -sum_{o in {A,B,C,D}} p_q(o) log_4 p_q(o).
• APES — range: [0, 1]
  • Average Pairwise Entropy Shift quantifies entropy variation across levels of a perturbation variable (e.g., gender or accent). Formula: APES_q^v = (2/(L(L-1))) * sum_{i<j} |H_q^{l_i} - H_q^{l_j}|.
• Fleiss’ Kappa — range: [-1, 1]
  • Measures categorical agreement across variable perturbations while correcting for chance. Formula: kappa = (P_bar - P_e) / (1 - P_e), where P_bar is average observed agreement and P_e is expected agreement.

Input / output format

Input: Concatenated audio segments containing a spoken question followed by four answer options (A, B, C, D), presented in either canonical or reversed order.

Output: A single selected option label (A, B, C, or D). The paper applies post-processing to correct formatting errors before scoring.

Scoring recipe

def score(predictions, gold, perturbations):
    # predictions: list of model outputs per perturbation condition
    # gold: ground truth option
    # perturbations: dict mapping variable -> list of levels
    
    # 1. Question Entropy (from model probs or one-hot predictions)
    H_q = -sum(p * log4(p) for p in probs)
    
    # 2. APES across variable levels
    H_levels = [compute_entropy(preds_for_level) for level in levels]
    APES = mean(abs(H_levels[i] - H_levels[j]) for i, j in pairs)
    
    # 3. Fleiss' Kappa across perturbations
    observed = agreement_rate(predictions)
    expected = sum(p**2 for p in overall_probs)
    kappa = (observed - expected) / (1 - expected)
    
    return H_q, APES, kappa

Common pitfalls

• Models require temperature=0 and specific post-processing to strip formatting artifacts before robustness analysis.
• Option order reversal changes the mapping of labels to content, so shifts must be computed on the underlying options, not the fixed A/B/C/D labels.
• Audio segments must be correctly concatenated according to the experimental condition before model inference.

Evidence (verbatim from paper)

To evaluate robustness under input perturbations, we employ three complementary metrics: entropy, APES, and Fleiss’ Kappa. These measures go beyond accuracy by assessing not only correctness but also the stability and consistency of model behavior.

Citation

@misc{wei2026biasinear,
  title={Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations},
  author={Wei et al. (2026)},
  year={2026},
  note={arXiv:2602.01030}
}

• arXiv: 2602.01030