Frill Noss Esc50 Eval

frill-noss-esc50-eval

FRILL: A Non-Semantic Speech Embedding for Mobile Devices — Peplinski et al. (2020) (arXiv:2011.04609, 2020)

What this evaluates

Evaluates the quality of lightweight, non-semantic speech embeddings by training simple downstream classifiers on averaged embedding features to perform audio classification tasks. It also measures inference latency on a mobile device to assess real-time suitability for on-device deployment.

Datasets

• NOSS benchmark — total ?; splits: train (-1), test (-1)
• ESC-50 (human sounds subset) — total ?; splits: train (-1), test (-1)
• Mask speech dataset — total 37000; splits: train (-1), test (-1)

Metrics

• test accuracy (primary) — range: [0, 1]
  • Accuracy = (number of correct predictions) / (total number of predictions). Reported as the best score across logistic regression, random forest, and LDA classifiers, with or without L2 speaker normalization.
• unweighted average recall — range: [0, 1]
  • Mean of recall scores computed per class, without weighting by class frequency. Used as an indicator for the mask speech task.

Input / output format

Input: Log-magnitude Mel spectrograms (64 bins, 25ms window, 10ms stride) derived from audio clips, or averaged per-utterance embedding vectors for downstream classification.

Output: Classification labels for downstream tasks; single embedding vector per utterance for latency/size benchmarking.

Scoring recipe

def evaluate(model, spectrograms, labels, dataset):
    embeddings = model(spectrograms)
    features = mean(embeddings, dim=time)
    if dataset in [SpeechCommands, CREMA-D, SAVEE]:
        features = l2_normalize(features, dim=speaker)
    best_acc = 0
    for clf in [LogisticRegression, RandomForest, LDA]:
        clf.fit(train_features, train_labels)
        preds = clf.predict(test_features)
        acc = accuracy_score(test_labels, preds)
        best_acc = max(best_acc, acc)
    return best_acc

Common pitfalls

• Embeddings are averaged over time per utterance, which discards temporal dynamics and may hurt performance on tasks requiring fine-grained temporal patterns.
• Speaker normalization (L2) is only applied to specific datasets (SpeechCommands, CREMA-D, SAVEE), not consistently across all benchmarks.
• Test labels for the mask speech task were unavailable at publication, so performance is reported as an indicator rather than a strict benchmark score.

Evidence (verbatim from paper)

We report the best test accuracy across combinations of downstream classifiers and normalization techniques. ... We use the first four published folds of ESC-50 for training, and the fifth for testing.

Citation

@misc{peplinski2020frill,
  title={FRILL: A Non-Semantic Speech Embedding for Mobile Devices},
  author={Peplinski et al. (2020)},
  year={2020},
  note={arXiv:2011.04609}
}

• arXiv: 2011.04609