Darf Srt Eval

darf-srt-eval

DARF: A data-reduced FADE version for simulations of speech recognition thresholds with real hearing aids — Hülsmeier et al. (2020) (arXiv:2007.05378, 2020)

What this evaluates

Evaluates the accuracy and computational efficiency of the DARF simulation framework for predicting speech recognition thresholds (SRT) in normal-hearing and hearing-impaired listeners across various acoustic maskers and hearing aid conditions.

Datasets

• Empirical SRT datasets (Hochmuth et al. 2015, Hülsmeier et al., Schädler et al. 2020a) — total ?; splits: test (-1)

Metrics

• SRT (primary) — range: dB SPL
  • Speech Recognition Threshold in dB SPL, representing the lowest signal-to-noise ratio at which speech is correctly identified.
• AST — range: other
  • Accuracy Speed Tradeoff, computed as simulation time multiplied by SRT error to balance computational cost and prediction accuracy.
• RMSE — range: dB
  • Root Mean Square Error between simulated and empirical SRTs.
• Bias — range: dB
  • Mean difference between simulated and empirical SRTs.
• R^2 — range: [0, 1]
  • Coefficient of determination measuring the proportion of variance in empirical SRTs explained by the simulation.

Input / output format

Input: Acoustic signals (speech and maskers like icra1m, icra5-250m, silence), hearing impairment profiles (e.g., N3), and hearing aid configurations/fittings.

Output: Simulated SRT values in dB SPL, along with derived metrics (AST, RMSE, Bias, R^2) comparing simulations to empirical data or FADE baselines.

Scoring recipe

def compute_ast(sim_time, srt_error):
    return sim_time * srt_error

def compute_rmse(pred_srt, emp_srt):
    return sqrt(mean((pred_srt - emp_srt)**2))

def compute_bias(pred_srt, emp_srt):
    return mean(pred_srt - emp_srt)

def compute_r2(pred_srt, emp_srt):
    ss_res = sum((emp_srt - pred_srt)**2)
    ss_tot = sum((emp_srt - mean(emp_srt))**2)
    return 1 - (ss_res / ss_tot)

Common pitfalls

• AST optimization depends heavily on the number of training vs. test sentences, with 120-240 training sentences yielding different tradeoffs.
• Differences between DARF and FADE vary significantly with masker type (stationary vs. fluctuating), requiring careful baseline selection.

Evidence (verbatim from paper)

To assess the tradeoff between simulation accuracy and time for the simulation, the Accuracy Speed Tradeoff (AST, Eq. [1]) was computed which is displayed in the lowest panel of Figure 5.

The simulations of both FADE versions correlate with an R2 of 0.99. However, the bias and RMSE indicate that hearing impairment might introduce an additional offset of about 1 dB from standard FADE: Both quantities exceeded the median offsets depicted in Fig. 6 by about 1 dB.

Citation

@misc{hulsmeier2020darf,
  title={DARF: A data-reduced FADE version for simulations of speech recognition thresholds with real hearing aids},
  author={Hülsmeier et al. (2020)},
  year={2020},
  note={arXiv:2007.05378}
}

• arXiv: 2007.05378