Voxknesset Demographic Eval

voxknesset-demographic-eval

VoxKnesset: A Large-Scale Longitudinal Hebrew Speech Dataset for Aging Speaker Modeling — Marmor et al. (2026) (arXiv:2603.01270, 2026)

What this evaluates

Evaluates whether the VoxKnesset dataset encodes meaningful demographic signals (gender, religion, birthplace) in pretrained speech representations. It probes the dataset's metadata quality and demographic coverage by training lightweight classifiers on extracted embeddings.

Datasets

• VoxKnesset Speaker-Attributed Longitudinal Subset — total ?; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed per demographic task to measure the Ridge classifier's discrimination ability between classes.

Input / output format

Input: 1024-dimensional mean-pooled WavLM-Large embeddings extracted from audio segments ≥30 seconds.

Output: Predicted demographic category and continuous confidence scores for each class.

Scoring recipe

for task in ['gender', 'religion', 'birthplace']:
    clf = RidgeClassifier()
    clf.fit(X_train, y_train)
    y_prob = clf.predict_proba(X_test)[:, 1]
    auc_scores[task] = roc_auc_score(y_test, y_prob)

Common pitfalls

• Splitting data by session instead of by speaker, which causes severe data leakage due to longitudinal overlap.
• Ignoring class imbalance in religion and birthplace categories, which can mislead accuracy-based evaluation but AUC remains robust.
• Using raw audio or different embedding models instead of the specified mean-pooled WavLM-Large features.

Evidence (verbatim from paper)

To validate that VoxKnesset captures meaningful speaker variation, we probe the demographic signal encoded in pretrained speech representations. We extract 1024-dimensional WavLM-Large embeddings (mean-pooled) and fit Ridge classifiers for gender, religion (4 classes), and birthplace (6 groups), see table 1.

Table 1: Demographic prediction with WavLM-L embeddings (Ridge). #C = classes; Maj. = majority baseline.

Citation

@misc{marmor2026voxknesset,
  title={VoxKnesset: A Large-Scale Longitudinal Hebrew Speech Dataset for Aging Speaker Modeling},
  author={Marmor et al. (2026)},
  year={2026},
  note={arXiv:2603.01270}
}

• arXiv: 2603.01270