Cs Kws Eval

cs-kws-eval

An Anchor-Free Detector for Continuous Speech Keyword Spotting — Zhao et al. (2022) (arXiv:2208.04622, 2022)

What this evaluates

Evaluates a model's ability to detect and localize multiple spoken keywords within continuous, untrimmed audio streams, distinguishing target keywords from background speech and silence.

Datasets

• LibriTop-20 — total ?; splits: train (-1), test (-1)
• CMAK-7 — total ?; splits: train (-1), test (-1)

Metrics

• 1D AP — range: [0, 1]
  • Average precision computed over the recall-precision curve using 1D Intersection over Union (IoU) between predicted and ground-truth keyword time intervals.
• mAP (primary) — range: [0, 1]
  • Mean average precision computed by averaging AP across all keyword classes, evaluated over IoU thresholds from 0.05 to 0.95 in steps of 0.05.
• FRR — range: [0, 1]
  • False Rejection Rate, measuring the proportion of ground-truth keywords that are missed by the detector at specified IoU thresholds (e.g., FRR@5, FRR@25).

Input / output format

Input: Fixed-length 1D audio segments or spectrograms (STFT). Shorter inputs are repeatedly padded; longer inputs are randomly cropped during training.

Output: Predicted keyword class, center location, and length (or start/end time) for each detected keyword instance.

Scoring recipe

def compute_mAP(preds, golds, iou_range=np.arange(0.05, 0.96, 0.05)):
    aps = []
    for cls in classes:
        cls_preds = [p for p in preds if p['class'] == cls]
        cls_golds = [g for g in golds if g['class'] == cls]
        tp, fp = [], []
        for pred in cls_preds:
            best_iou = max(iou_1d(pred, g) for g in cls_golds)
            if best_iou >= 0.05:
                tp.append(best_iou)
            else:
                fp.append(0)
        ap = compute_ap_from_tp_fp(tp, fp)
        aps.append(ap)
    return np.mean(aps)

def iou_1d(pred, gt):
    inter = max(0, min(pred.end, gt.end) - max(pred.start, gt.start))
    union = (pred.end - pred.start) + (gt.end - gt.start) - inter
    return inter / union if union > 0 else 0

Common pitfalls

• Uses IoU range [0.05, 0.95] instead of the standard [0.5, 0.95] used in visual detection, reflecting higher temporal tolerance in speech.
• Requires an auxiliary 'unknown' class to filter out interfering words and silence; omitting it significantly degrades performance.
• Sliding-window classifiers adapted for this task suffer from low temporal resolution compared to anchor-free regression, leading to artificially low AP scores.

Evidence (verbatim from paper)

CSKWS can borrow the evaluation metrics from these related tasks. However, there are more keywords in CSKWS and they appear more frequently than in trigger words detection. Besides, CSKWS is essentially a detection task instead of classification task, so we propose to evaluate its solutions with object detection metrics known as 1D AP(average precision) and mAP (mean average precision). 1D IoU (Intersection over Union) is computed for each detection result with respect to the groundtruth, and mAP is computed in an IoU range of (0.05,0.95) by step 0.05.

Citation

@misc{zhao2022anchorfree,
  title={An Anchor-Free Detector for Continuous Speech Keyword Spotting},
  author={Zhao et al. (2022)},
  year={2022},
  note={arXiv:2208.04622}
}

• arXiv: 2208.04622