Backdoormbti Eval

backdoormbti-eval

BackdoorMBTI: A Backdoor Learning Multimodal Benchmark Tool Kit for Backdoor Defense Evaluation — Yu et al. (2024) (arXiv:2411.11006, 2024)

What this evaluates

This benchmark evaluates the robustness and effectiveness of multimodal backdoor attacks and defense mechanisms across image, text, and audio modalities. It specifically probes how well defenses maintain clean accuracy while suppressing attack success rates under varying noise conditions and label corruption.

Datasets

• CIFAR-10 — total ?; splits: test (-1)
• SST-2 — total ?; splits: test (-1)
• SpeechCommands — total ?; splits: test (-1)

Metrics

• ASR (primary) — range: [0, 1]
  • Attack Success Rate: the fraction of poisoned inputs that are misclassified to the attacker-specified target label. Calculated as the number of poisoned samples predicted as the target class divided by the total number of poisoned samples.
• accuracy (primary) — range: [0, 1]
  • Clean Accuracy: the fraction of clean (unpoisoned) inputs that are correctly classified by the model. Calculated as the number of correctly predicted clean samples divided by the total number of clean samples.
• recall — range: [0, 1]
  • Detection Recall: the fraction of poisoned samples correctly identified as malicious by a defense method.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for the defense detection task.
• DAC — range: [0, 1]
  • Detection Accuracy: the overall fraction of samples (clean and poisoned) correctly classified by the defense mechanism.

Input / output format

Input: Multimodal inputs (images, text sequences, audio waveforms) processed through modality-specific backbones (ResNet, BERT, CNN). Inputs may be clean, poisoned with triggers, or corrupted with Gaussian/text noise and label flips.

Output: Class predictions (hard labels) for the victim model, and binary detection flags (clean/malicious) for defense methods.

Scoring recipe

def compute_asr(predictions, labels, target_label):
    correct = sum(1 for p, l in zip(predictions, labels) if p == target_label)
    return correct / len(labels) if len(labels) > 0 else 0.0

def compute_accuracy(predictions, labels):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    return correct / len(labels) if len(labels) > 0 else 0.0

Common pitfalls

• Confusing ASR with accuracy: high ASR indicates a successful attack, whereas effective defenses require low ASR and high accuracy.
• Ignoring modality-specific architectural constraints: pruning-based defenses (FP, CLP) fail on text models like BERT because they rely on batch normalization layers absent in transformers.
• Overlooking fixed noise simulation parameters: the benchmark uses specific settings (Gaussian noise mean=0, var=1; text CER=0.1; 25% label noise) which may not generalize to all real-world corruption levels.

Evidence (verbatim from paper)

Figure 4 illustrates the relationship between accuracy and ASR of backdoor defenses, with effective methods typically positioned in the top-left corner, indicating high accuracy and low ASR on sanitized models.

Citation

@misc{yu2024backdoormbti,
  title={BackdoorMBTI: A Backdoor Learning Multimodal Benchmark Tool Kit for Backdoor Defense Evaluation},
  author={Yu et al. (2024)},
  year={2024},
  note={arXiv:2411.11006}
}

• arXiv: 2411.11006