Anti False Positive Decision Making

Anti-False-Positive Decision Making

When to Use

• Calibrate when to act vs monitor when false positives block users, delay revenue, or burn analyst capacity
• Design evidence bars and multi-signal corroboration before irreversible actions (block, freeze, terminate, auto-remediate)
• Tune security, fraud, or compliance alerts with explicit FP/FN trade-offs and base-rate context
• Define tiered response paths and human-in-the-loop gates for screening and monitoring workflows
• Choose metrics (precision, recall, FDR) and review cadence for alert quality—not model training alone
• Document decision rationale for auditors, regulators, or post-incident review when disposition matters
• Reduce alert fatigue and over-blocking without silently accepting unacceptable false negatives

When NOT to Use

• Train, deploy, or optimize production ML classifiers (feature engineering, hyperparameters, MLOps) → data-scientist, ml-ops-engineer
• Legal conclusions, sanctions determinations, or contract interpretation → commercial-counsel
• Author detection rules or queries only with no decision policy, escalation, or disposition framework → information-security-engineer, defensive-security-analyst, aml-compliance (as primary)
• Generic strategy stress-tests without operational decision design → executive-mentor
• Full AML program design, KYC policy, or STR narrative drafting → aml-compliance, str-report
• Internal audit workpapers and control effectiveness sampling → auditor
• Pre-flight architecture or build validation without alert/disposition lens → build-validator

Related skills

Core Workflows

1. Frame the decision and costs

1. Name the action (block, freeze, escalate, auto-close, notify, quarantine)
2. List cost of false positive (customer harm, ops load, revenue, reputation, legal exposure from over-action)
3. List cost of false negative (fraud loss, breach, regulatory miss, safety)
4. Estimate base rate or prevalence band for the population (see reference)
5. Set explicit risk appetite for this decision class (who approves exceptions)

See references/anti_fp_decision_scope.md for scope, boundaries, and handoffs.

2. Set evidence bar and corroboration

signal(s) → minimum evidence tier → optional corroboration → disposition → audit record

• Define single-signal vs multi-signal requirements before irreversible action
• Prefer independent evidence types (rule + graph + human review; not two copies of same feature)
• Document what would falsify the hypothesis before closing as benign

See references/evidence_bars_and_corroboration.md.

3. Map FP/FN trade-offs and thresholds

• Plot operating points: stricter threshold → fewer FPs, more FNs (usually)
• Use base rate to interpret raw alert rates; avoid judging rules on volume alone
• Segment by customer tier, geography, product, or asset criticality where policy differs

See references/fp_fn_tradeoffs_and_base_rates.md.

4. Design tiered response and HITL gates

See references/tiered_response_and_hitl_gates.md.

5. Tune alerts in security and compliance contexts

• Govern threshold changes: hypothesis, backtest, approval, post-change review
• Separate detection quality from workflow quality (routing, SLAs, training)
• Track precision at disposition and time-to-benign-close, not alert count alone

See references/security_compliance_alert_tuning.md.

6. Measure, calibrate, and document

• Define metrics owners and review cadence (weekly ops, quarterly risk)
• Record rationale template on every material disposition
• Run calibration reviews: sample closed-as-benign and closed-as-true-positive

See references/metrics_calibration_and_documentation.md.

When to load references