Multiqt Question Tracking Eval

multiqt-question-tracking-eval

MultiQT: Multimodal Learning for Real-Time Question Tracking in Speech — Havtorn et al. (2020) (arXiv:2005.00812, 2020)

What this evaluates

Evaluates a model's ability to perform real-time, multimodal sequence labeling to detect and classify questions in emergency call speech. It probes robustness to noisy ASR transcriptions and temporal alignment under streaming conditions.

Datasets

• question and symptoms tracking datasets — total ?; splits: 5-fold cross-validation (-1)

Metrics

• TIMESTEP F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed per timestep across all classes, then micro-averaged over examples. Captures exact temporal alignment of question segments and is sensitive to misalignment.
• INSTANCE F1 — range: [0, 1]
  • Macro-averaged F1 score that counts a prediction correct if there are at least five consecutive correctly labeled timesteps within a sequence. Explicitly excludes the non-question label.

Input / output format

Input: Streaming audio encoded as 40 log-mel features (0.02s window, 0.01s stride) concatenated with real-time ASR character-level transcriptions (stride 2 relative to audio).

Output: Per-timestep categorical label prediction (e.g., question type or non-question).

Scoring recipe

def compute_metrics(preds, golds):
    # preds, golds: list of label sequences per timestep
    # TIMESTEP F1: compare pred vs gold per timestep, micro-average over examples
    tp, fp, fn = 0, 0, 0
    for p, g in zip(preds, golds):
        for pi, gi in zip(p, g):
            if pi == gi: tp += 1
            elif pi != gi:
                fp += 1
                fn += 1
    timestep_f1 = 2*tp/(2*tp+fp+fn)
    
    # INSTANCE F1: exclude non-question, check for 5 consecutive correct
    instance_correct = 0
    for p, g in zip(preds, golds):
        valid_p, valid_g = [], []
        for pi, gi in zip(p, g):
            if gi != 'non_question': valid_p.append(pi); valid_g.append(gi)
        if has_k_consecutive(valid_p, valid_g, k=5): instance_correct += 1
    instance_f1 = macro_f1(valid_p, valid_g) # macro over classes
    return timestep_f1, instance_f1

Common pitfalls

• TIMESTEP metric is highly sensitive to temporal misalignment between audio and ASR output.
• INSTANCE metric explicitly excludes the non-question label, which can inflate scores on imbalanced data.
• Baseline models (RF/FNN) are trained on fixed segments but evaluated in a streaming/sliding-window fashion, causing a significant performance drop not seen in MultiQT.

Evidence (verbatim from paper)

For each model we report two F1 scores with respective precisions and recalls macroaveraged over the classes. - TIMESTEP: For each timestep, the model prediction is compared to the gold label. The metrics are computed per timestep and micro-averaged over the examples. This metric captures the model performance in finding and correctly classifying entire audio segments that represent questions and is sensitive to any misalignment. -INSTANCE: A more forgiving metric which captures if sequences of the same label are found and correctly classified with acceptance of misalignment. Here, the prediction counts as correct if there are at least five consecutive correctly labeled time steps within the sequence, as a heuristic to avoid ambiguity between classes. This metric also excludes the non-question label.

Citation

@misc{havtorn2020multiqt,
  title={MultiQT: Multimodal Learning for Real-Time Question Tracking in Speech},
  author={Havtorn et al. (2020)},
  year={2020},
  note={arXiv:2005.00812}
}

• arXiv: 2005.00812