Vigil Cognitive Bias Eval

vigil-cognitive-bias-eval

VIGIL: An Extensible System for Real-Time Detection and Mitigation of Cognitive Bias Triggers — Kang et al. (2026) (arXiv:2604.03261, 2026)

What this evaluates

Evaluates a real-time browser extension's ability to detect and mitigate cognitive bias triggers in online text. It probes span-level identification of persuasive rhetoric and bias patterns, alongside system latency and mitigation quality.

Datasets

• SemEval-2020 Task 11 — total ?; splits: test (-1)
• Moralization Corpus — total ?; splits: test (-1)

Metrics

• micro-F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, weighted by the number of true instances for each class.
• macro-F1 — range: [0, 1]
  • Harmonic mean of precision and recall, calculated independently for each class and then averaged unweighted.
• median latency — range: seconds
  • Median time in seconds to process a single text span through the detection pipeline.

Input / output format

Input: Text spans from online content (e.g., Twitter/X posts, news articles) presented in real-time as the user scrolls.

Output: Span-level detection labels indicating the presence of specific cognitive bias triggers, followed by optional LLM-generated neutralized/reformulated text.

Scoring recipe

def compute_f1(predictions, gold, average='micro'):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    if average == 'micro':
        return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    else:
        return (precision + recall) / 2

Common pitfalls

• The SemEval-2020 evaluation strictly follows the protocol from Sprenkamp et al. [10], not the original SemEval guidelines.
• Latency measurements are highly backend-dependent (regex: 0.03 ms, WebGPU: 3.4 s, cloud: 3.9 s), making cross-system comparisons invalid without specifying the inference tier.
• The system deliberately optimizes for precision over recall on the SemEval benchmark, which may artificially suppress the reported F1 score.

Evidence (verbatim from paper)

On SemEval-2020 Task 11 [3] (using the protocol from Sprenkamp et al. [10]), the production prompt achieves a very competitive micro-F1 = 0.533 with precision $= 0.626$ , deliberately favoring precision. The moralization plugin achieves macro-F1 = 0.789 on the Moralization Corpus [1], competitive with the corpus authors' best (0.772).

Citation

@misc{kang2026vigil,
  title={VIGIL: An Extensible System for Real-Time Detection and Mitigation of Cognitive Bias Triggers},
  author={Kang et al. (2026)},
  year={2026},
  note={arXiv:2604.03261}
}

• arXiv: 2604.03261