Deduplicator (/mantis-dedupe)
System Goal
Duplicate Finding Merger. Evaluates lists of raw findings to cluster and
consolidate identical or highly overlapping issues into singular, descriptive
records.
Command Definition
- Command:
/mantis-dedupe
- Description: Consolidates raw security findings to eliminate redundant
reports.
- Arguments (optional; supplied by the orchestrator, consumed by Block A):
--snapshot_root/--snapshot_id/--state_root. All absent ->
MODE-OFF/legacy mode (behaves as today; snapshot gating disabled).
Input/Output Contract
- Reads:
workspace/findings/ (raw finding JSON files, ignoring .trash/).
workspace/archive/findings_pass_*/*.json and
workspace/archive/loop*_findings/*.json (to skip findings already
evaluated and triaged in previous passes).
workspace/.mantis_state.json (to track current loop pass).
- Writes:
- Moves duplicate findings to
workspace/findings/.trash/ after setting
"status": "DUPLICATE" and "duplicate_of".
- Sets
"possible_duplicate_of" (soft, non-terminal) on NOT_MATCHED matches
and stamps "discovery_commit" on current findings that lack it. Reads
active_snapshot/snapshot_pinned from .mantis_state.json.
- Appends transaction logs to
workspace/.tx_log.jsonl.
- Generates/executes merging script
workspace/helpers/merge_findings.py.
- Updates primary finding
workspace/findings/<primary_id>.json (merges
fields and history).
- Preconditions:
workspace/findings/ must exist and contain finding files.
- Idempotency Guarantee:
- Cross-references against archived findings in
workspace/archive/ to filter
out any findings already processed in previous passes of this run.
Snapshot-gated: a resolved archived finding on a differing snapshot is
flagged as POSSIBLE REGRESSION (never silently filtered). Logs transactions
to workspace/.tx_log.jsonl to support tracking and potential rollbacks.
Deterministic merging rules implemented in merge_findings.py.
Instructions
Step 0: Locator Resolution (run first)
LOCATOR RESOLUTION (before reading ANY target code or artifact):
0. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
state for provenance/annotation; NEVER stop merely because a code root is unset.
1. Determine CODE_ROOT, in this priority order:
a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
(used when a caller hands you a prepared tree, e.g. a patched shadow).
b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
c. Else read state_root/workspace/.mantis_state.json (state_root from
--state_root if passed, else ./workspace/... relative to the current dir)
-> active_snapshot.root / .snapshot_id / .snapshot_pinned.
d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
2. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
deliberately mutated and is sentinel-EXEMPT.)
3. PATH FIELDS:
- SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
NOT of the form <existing-path>:<integer> is a non-source LOCATOR
(symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
line-range and line-existence logic.
- STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
kb_references, repro_file_path, reattack_file_path, helper scripts, report
files, and all state/findings JSON.
4. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
(mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
cd into CODE_ROOT.
5. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
(the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
lacks .git/.hg/.repo.
6. Every shell command uses ABSOLUTE paths and sets its own working directory on
that call. Do NOT assume the working directory persists between calls.
[!NOTE] CURRENT-PASS CHECK (defensive; the binding guarantee is on the
harness per mantis-pipeline-adapter Scenario 2): if active_snapshot is
present AND active_snapshot.pass != state.pass_number, treat the snapshot as
STALE for this pass — STOP "stale active_snapshot: pass mismatch" or degrade
as HALT (snapshot_pinned effectively false: no authoritative verdicts, Block
B NOT_MATCHED, reproduce not_attempted). This catches a custom harness that
preserved active_snapshot across the Stage 15 pass increment without
re-pinning. The reference meta-agent re-pins every pass, so this check never
fires there. Block B itself cannot detect this (it is snapshot_id-only, not
pass-aware).
Notes: workspace/findings/, workspace/archive/, workspace/.tx_log.jsonl,
workspace/helpers/ and .mantis_state.json are STATE-RELATIVE (under
--state_root). Any code snippet you inspect for a finding is SNAPSHOT-RELATIVE
(under CODE_ROOT). Never write under CODE_ROOT.
Review a list of security findings and merge duplicate findings that refer to
the exact same security flaw or adjacent code paths.
Execute your task as follows:
Load Raw Findings & Archived Findings Queue:
- List the contents of the directory and read the files in
workspace/findings/. If the directory is empty or does not exist, notify
the user and exit.
- Important: Ignore hidden files and directories (such as the
.trash/
subdirectory) when listing or processing findings.
- Locate and load all archived finding JSON files from previous loop passes,
if they exist, under
workspace/archive/findings_pass_*/*.json and
workspace/archive/loop*_findings/*.json. These files represent
vulnerabilities that have already been fully evaluated, triaged, and
potentially patched in previous passes.
- Important: Do NOT read or deduplicate against
workspace/historical_learnings.jsonl (VCS history), as we want to catch
regressions if old bugs were reintroduced.
Filter Loop Duplicates (snapshot-gated). First, stamp discovery_commit
on any CURRENT finding that lacks it, using active_snapshot.snapshot_id
from .mantis_state.json (skip when unpinned). Then, for each current
finding that matches an archived finding (by code_paths+title
similarity), run:
Signature-based candidate matching (Phase 3) — TIGHTENS, never replaces:
signature may only PROMOTE a pair to "candidate for the pairwise snapshot
check"; it may NEVER by itself cause a hard DUPLICATE/trash. A pair is a
candidate for the Pairwise Snapshot Match Check below ONLY if it satisfies
BOTH:
- it matches under today's
code_paths + title similarity, comparing
code_paths entries line-inclusively (WITH their trailing :line); AND
- (when both findings have a
signature) their signature fields are equal.
A signature match WITHOUT the code_paths + title agreement is NOT a
duplicate — at most a soft possible_duplicate_of (keep the finding
ACTIVE), never a trash. Rationale: signature strips the line number and
all-but-first code_paths entry, so two DISTINCT bugs in the same file
(e.g. parser.c:100 vs parser.c:900) with the same title+CWE share one
signature; trashing on signature alone would silently delete a real
finding. If EITHER lacks signature, use today's code_paths + title
similarity matching unchanged.
PAIRWISE SNAPSHOT MATCH CHECK (decides MATCHED vs NOT_MATCHED) — compares the
CURRENT finding's discovery_commit against the ARCHIVED finding's
discovery_commit for this pair (NOT against SNAPSHOT_ID):
- If
snapshot_pinned is false AND there is NO active_snapshot in state
(MODE-OFF) -> NOT_MATCHED. Stop. (In HALT — active_snapshot present but
snapshot_pinned=false — do NOT short-circuit here; fall through to the
pairwise comparison below, which will be NOT_MATCHED because the current
finding's discovery_commit is a live: id that will not equal the
archived one.)
- Read the CURRENT finding's
discovery_commit and the ARCHIVED finding's
discovery_commit:
- If EITHER is missing, empty, or the literal
"MIXED" -> NOT_MATCHED.
- If they are NOT byte-for-byte equal to each other -> NOT_MATCHED.
- If they ARE byte-for-byte equal to each other (both present, non-MIXED)
-> MATCHED. There is no other route to MATCHED; never fuzzy-compare. The
global "default the field and proceed" backward-compat rule does NOT
apply to
discovery_commit: absent = NOT_MATCHED. (There is NO separate
"dirty" gate: a dirty tree's SNAPSHOT_ID already embeds the working-tree
content hash, so within-pass findings MATCH and cross-pass bare-commit
findings do not.) Note: this is a PAIRWISE check (current vs archived),
NOT a check against the global SNAPSHOT_ID — dedupe stamps the current
finding's discovery_commit to SNAPSHOT_ID in Step 2 above, so a
check against SNAPSHOT_ID would always be MATCHED and would trash
reintroduced/regression bugs as DUPLICATE.
Then, using the idempotency rule (Input/Output Contract → Idempotency
Guarantee) to avoid double-writes, decide mechanically:
- MATCHED (both present and equal): soft-delete the current finding as a
loop-duplicate exactly as before — set
"status": "DUPLICATE" and
"duplicate_of": "<archived_uuid>", clear possible_duplicate_of if
present, ensure mkdir -p workspace/findings/.trash/, move it there, and
log a loop_filter transaction in workspace/.tx_log.jsonl. If the
current finding lacks lineage_id but the archived finding has one,
inherit the archived finding's lineage_id onto the current finding before
moving it (so the lineage chain is preserved across the merge).
- NOT_MATCHED (differ, or either absent): do NOT set
DUPLICATE and do
NOT move to trash. Keep the current finding ACTIVE and set
"possible_duplicate_of": "<archived_uuid>" (a soft, non-terminal hint).
If the current finding lacks lineage_id but the archived finding has one,
inherit the archived finding's lineage_id onto the current finding (so
the lineage chain is preserved for report folding even when the findings
are on different snapshots).
[!IMPORTANT] STATUS & DUPLICATE INVARIANTS:
- A finding MUST NOT carry both
duplicate_of and possible_duplicate_of
pointing to the same target UUID.
status = "DUPLICATE" MUST NOT coexist with possible_duplicate_of
pointing to the same target UUID.
- Under NOT_MATCHED (outside the MODE-OFF fallback exception), the
finding's
status MUST remain active (e.g. VALID,
PROVISIONALLY_VALID, NEEDS_RESEARCH), duplicate_of MUST NOT be set,
and the finding MUST NOT be moved to .trash/. Setting
possible_duplicate_of is a non-terminal hint only.
- POSSIBLE REGRESSION: if the archived match has a RESOLVED status
(
patch_status in {VERIFIED_SECURE,MITIGATION_PROPOSED} OR
status==FALSE_POSITIVE OR production_viability==NON_VIABLE) AND the
pair is NOT MATCHED, keep the current finding ACTIVE, add a history note
"POSSIBLE REGRESSION vs <archived_uuid>", and do NOT filter it. (A
reverted fix re-discovered on new code must never be trashed.)
- EXACT-UUID retry exception (unchanged): if the current finding has the
EXACT SAME UUID as the archived one, it was intentionally copied back for a
retry — do NOT filter it (keep as-is).
- Permanently-unpinned exception (MODE-OFF only — no
active_snapshot):
if there is NO active_snapshot in state (MODE-OFF = today's default; a
target with no snapshot boundary, e.g. a live endpoint), snapshot_pinned
is false at the PASS level (active_snapshot.snapshot_pinned — it is NOT a
per-finding field) and Block B is uninformative — fall back to today's
dedup by signature if present, else stable_key = normalized_title +
first code_paths entry including its trailing :line
(line-inclusive, same as Step 2). Rationale: stripping :line would
collapse two DISTINCT bugs in the same file (e.g. parser.c:100 vs
parser.c:900) with the same title+CWE into one — silently deleting a real
finding. Note: signature itself strips :line by design (it is a coarse
identity for cross-pass lineage, not a dedup key); this fallback therefore
prefers signature only when stable_key's line-inclusive match ALSO
agrees, never on signature alone. This preserves dedup for targets that
can never MATCH. When active_snapshot IS present but unpinned (HALT
mode), this exception does NOT fire: keep the snapshot-gated behavior above
(NOT_MATCHED → keep ACTIVE + possible_duplicate_of, never DUPLICATE).
POSSIBLE REGRESSION takes precedence over this fallback: a
resolved-archived finding paired with a NOT_MATCHED current finding is
ALWAYS kept active (never trashed), regardless of mode.
3-Tier Deduplication Ladder (Deterministic & Semantic Matching): When
resolving finding lineages and evaluating deduplication candidates, the
deduplicator uses a hierarchical 3-tier ladder designed for sub-millisecond
fast-path execution with semantic RAG fallback:
Tier 1 (Fast-Path Exact Heuristic Anchors — < 1ms, 0 tokens):
- Exact content identity signature match:
hashlib.sha256(canonical_fp | canonical_cwe | target_symbol)
(invariant to line shifts, backticks, and title paraphrasing).
- Exact
canonical_filepath + normalized CWE + target_symbol match.
- Strict line proximity window ($\le 3$ lines) on exact canonical
filepath when target symbol is empty.
- Result: If matched, immediately inherit ancestor
lineage_id without
LLM or embedding overhead.
Tier 2 (RCA Normalization): If Tier 1 heuristic matching does not
produce an exact anchor, synthesize a standardized 5-line Root Cause
Analysis (RCA) summary:
Component: Normalized canonical filepath and symbol.
Vulnerability Class: Canonical CWE taxonomy identifier and name.
Root Cause Mechanism: Underlying programming or logic defect.
Failure Condition: Specific input, state, or boundary condition.
Taint Dataflow: Source-to-sink dataflow trajectory.
Tier 3 (Vector Embedding & Cosine Similarity Scan):
- Project the standardized RCA summary into dense vector embeddings using
the configured multi-provider embedding engine (default:
vertex_ai/gemini-embedding-001).
- Perform nearest-neighbor scan over historical
lineage_vectors in the
database using bounded cosine similarity.
- CWE Family & Class Structural Guard: When comparing query finding
vectors against candidate lineage records, if both findings have explicit
CWE classifications (e.g. CWE-89 vs CWE-78), normalize them. If they
belong to distinct, incompatible CWE IDs, skip candidate vector
comparison entirely to structurally prevent false-merging distinct
vulnerability classes regardless of cosine score.
- Positive threshold ($\ge 0.90$): Semantically equivalent findings
with differing phrasing, scanner labels, or line shifts merge into the
same ancestor
lineage_id (default: 0.90, configurable via
EMBEDDING_SIMILARITY_THRESHOLD).
- Negative threshold ($< 0.70$): Distinct vulnerability classes (e.g.,
SQLi vs Command Injection, Stored vs Reflected XSS) maintain low
similarity and fail closed, minting a fresh unique
lineage_id.
Filter Duplicate Findings in Current Batch: Check the current findings
against each other to find duplicates. Two findings are duplicates ONLY if
they share the same code_paths entry line-inclusively (WITH trailing
:line) AND have the same or highly similar title. If multiple findings
refer to the exact same flaw at the same location, they must be merged.
Findings at different lines in the same file are DISTINCT — never merge them.
Map/Reduce Chunking Strategy (For Scale): If there are many finding files
(e.g., > 20 items), use a Map/Reduce approach to group them by target file or
component before checking for overlaps to avoid context window limits.
Token-Optimized Consolidation and Merging: To minimize LLM output tokens
and prevent data loss, do not manually rewrite or output the merged JSON
files in your response. Instead, follow this pattern:
- Identify Duplicates: Internally map which findings are duplicates of a
primary finding.
- Reusable Deterministic Scripting (versioned): Write a reusable helper
script (e.g.
workspace/helpers/merge_findings.py) whose FIRST LINE is
exactly # MANTIS_HELPER_VERSION = 2. Before reusing an existing helper,
grep its first lines for MANTIS_HELPER_VERSION = 2; if that marker is
absent or a different integer (a helper left by an older pipeline
version), REGENERATE the helper. Only reuse it when the marker matches.
The script must follow these deterministic rules:
- Title: Pick the most comprehensive and descriptive title.
- ID: Preserve the unique
"id" of the primary finding being kept.
- Severity: Pick the highest severity level specified among the merged
items.
- Privileges Required: Inherit the most severe privilege requirement
(priority:
NONE > LOW > HIGH).
- Attacker Position: Inherit the most critical position requirement
(priority:
EXTERNAL > INTERNAL_NETWORK > IN_CLUSTER > LOCAL >
HOST_SYSTEM > SUPPLY_CHAIN > PHYSICAL_TEMPORARY >
PHYSICAL_LONG_TERM).
- User Interaction: Inherit the most severe user interaction
requirement (priority:
NONE > REQUIRED).
- Code Paths: Collect and deduplicate all file paths and line numbers
into a single unique array.
- Description, Mitigation, & Impact: Concatenate cleanly.
- History: Concatenate and preserve all
"history" entries from the
merged findings. Append a new entry to the "history" array for this
merge action conforming to the schema (containing "stage": "dedupe",
"action": "merge",
"details": "Merged duplicate findings: [comma-separated-ids]",
"pass_number": <current_pass_number>, and
"timestamp": "<current_iso8601_timestamp>").
- Unknown keys & provenance (MANDATORY): The script MUST copy through
EVERY key it does not explicitly handle (including
discovery_commit,
repro_snapshot_id, patch_base_snapshot, possible_duplicate_of,
signature, lineage_id, cwe) from the primary finding onto the
merged object — never drop unknown fields. It MUST REFUSE to merge two
findings whose discovery_commit values differ (they describe different
code versions); leave them separate and log the refusal. When merging
findings that all share one discovery_commit, preserve it unchanged.
- Execute the Script: Run your script to update the primary finding's
file (
workspace/findings/<primary_id>.json) on disk.
Transactional Staged Clean Up: Do not permanently delete redundant files.
Ensure the trash directory exists (e.g.,
mkdir -p workspace/findings/.trash/). Before moving, the script must update
the duplicate finding files, setting "status": "DUPLICATE" and
"duplicate_of": "<primary_uuid>". Move the merged duplicate .json files
to the trash staging directory (workspace/findings/.trash/). For every file
moved, append a transaction record to workspace/.tx_log.jsonl.
Transaction Log Schema Format (workspace/.tx_log.jsonl)
Each line must be a self-contained JSON object documenting the transaction:
{"timestamp": "2026-07-14T15:13:00Z", "action": "loop_filter | dedupe_merge", "primary_uuid": "[UUID] (or null for loop_filter)", "moved_uuid": "[UUID]"}
This cleans up the directory for downstream stages while preserving rollback
capability.
When complete, notify the user.
1---
2name: mantis-dedupe
3description: >-
4 Consolidates raw security findings to eliminate redundant reports.
5 Use when raw findings have been generated by the researcher and need consolidation before review.
6 Don't use for initial code auditing or patch generation.
7metadata:
8 source: https://skillmd.com/skills/google/mantis-dedupe
9---
10
11# Deduplicator (/mantis-dedupe)
12
13## System Goal
14
15Duplicate Finding Merger. Evaluates lists of raw findings to cluster and
16consolidate identical or highly overlapping issues into singular, descriptive
17records.
18
19## Command Definition
20
21- **Command:** `/mantis-dedupe`
22- **Description:** Consolidates raw security findings to eliminate redundant
23 reports.
24- **Arguments (optional; supplied by the orchestrator, consumed by Block A):**
25 `--snapshot_root`/`--snapshot_id`/`--state_root`. All absent ->
26 MODE-OFF/legacy mode (behaves as today; snapshot gating disabled).
27
28## Input/Output Contract
29
30- **Reads**:
31 - `workspace/findings/` (raw finding JSON files, ignoring `.trash/`).
32 - `workspace/archive/findings_pass_*/*.json` and
33 `workspace/archive/loop*_findings/*.json` (to skip findings already
34 evaluated and triaged in previous passes).
35 - `workspace/.mantis_state.json` (to track current loop pass).
36- **Writes**:
37 - Moves duplicate findings to `workspace/findings/.trash/` after setting
38 `"status": "DUPLICATE"` and `"duplicate_of"`.
39 - Sets `"possible_duplicate_of"` (soft, non-terminal) on NOT_MATCHED matches
40 and stamps `"discovery_commit"` on current findings that lack it. Reads
41 `active_snapshot`/`snapshot_pinned` from `.mantis_state.json`.
42 - Appends transaction logs to `workspace/.tx_log.jsonl`.
43 - Generates/executes merging script `workspace/helpers/merge_findings.py`.
44 - Updates primary finding `workspace/findings/<primary_id>.json` (merges
45 fields and history).
46- **Preconditions**:
47 - `workspace/findings/` must exist and contain finding files.
48- **Idempotency Guarantee**:
49 - Cross-references against archived findings in `workspace/archive/` to filter
50 out any findings already processed in previous passes of this run.
51 Snapshot-gated: a resolved archived finding on a differing snapshot is
52 flagged as POSSIBLE REGRESSION (never silently filtered). Logs transactions
53 to `workspace/.tx_log.jsonl` to support tracking and potential rollbacks.
54 Deterministic merging rules implemented in `merge_findings.py`.
55
56## Instructions
57
58### Step 0: Locator Resolution (run first)
59
60```
61LOCATOR RESOLUTION (before reading ANY target code or artifact):
620. ROLE: If this skill NEVER reads target source (report, calibrate, reflect),
63 you are a FINDINGS-ONLY stage: skip steps 2-6; still read active_snapshot from
64 state for provenance/annotation; NEVER stop merely because a code root is unset.
651. Determine CODE_ROOT, in this priority order:
66 a. If --target_root is passed on THIS invocation, CODE_ROOT = --target_root.
67 It is AUTHORITATIVE and OVERRIDES SNAPSHOT_ROOT and the state fallback
68 (used when a caller hands you a prepared tree, e.g. a patched shadow).
69 b. Else if --snapshot_root (or SNAPSHOT_ROOT) is passed, use it.
70 c. Else read state_root/workspace/.mantis_state.json (state_root from
71 --state_root if passed, else ./workspace/... relative to the current dir)
72 -> active_snapshot.root / .snapshot_id / .snapshot_pinned.
73 d. Else (no arg AND no readable active_snapshot): CODE_ROOT = current directory,
74 treat snapshot_pinned = false (MODE-OFF). Do NOT stop.
752. SENTINEL CHECK (only if snapshot_pinned is true AND you did NOT take path 1a):
76 verify CODE_ROOT/.mantis_snapshot_id exists and equals SNAPSHOT_ID. If missing
77 or different -> STOP "snapshot sentinel mismatch". (A --target_root tree (1a) is
78 deliberately mutated and is sentinel-EXEMPT.)
793. PATH FIELDS:
80 - SNAPSHOT-RELATIVE (read under CODE_ROOT): code_paths entries; plan target_files
81 that are file paths. Strip ONLY a trailing ":<digits>". A code_paths entry
82 containing "://" is a URL/endpoint, NOT a file read. A code_paths entry that is
83 NOT of the form <existing-path>:<integer> is a non-source LOCATOR
84 (symbol/offset/endpoint): only check that the artifact/symbol exists; skip ALL
85 line-range and line-existence logic.
86 - STATE-RELATIVE (read/write under state_root/workspace, NEVER prefix CODE_ROOT):
87 kb_references, repro_file_path, reattack_file_path, helper scripts, report
88 files, and all state/findings JSON.
894. Never WRITE under CODE_ROOT when snapshot_pinned is true. Any command that
90 compiles, generates, or writes artifacts MUST run in a PRIVATE SHADOW copy
91 (mktemp -d from CODE_ROOT), never with cwd=CODE_ROOT. Read-only inspection may
92 cd into CODE_ROOT.
935. VCS-METADATA CARVE-OUT: history-log extraction and any VCS diff/blame command
94 run in the LIVE repository root (which still has .git/.hg/.repo), NOT CODE_ROOT
95 (the snapshot copy strips VCS metadata). Do NOT stop merely because CODE_ROOT
96 lacks .git/.hg/.repo.
976. Every shell command uses ABSOLUTE paths and sets its own working directory on
98 that call. Do NOT assume the working directory persists between calls.
99```
100
101> [!NOTE] **CURRENT-PASS CHECK (defensive; the binding guarantee is on the
102> harness per `mantis-pipeline-adapter` Scenario 2):** if `active_snapshot` is
103> present AND `active_snapshot.pass != state.pass_number`, treat the snapshot as
104> STALE for this pass — STOP "stale active_snapshot: pass mismatch" or degrade
105> as HALT (`snapshot_pinned` effectively false: no authoritative verdicts, Block
106> B NOT_MATCHED, reproduce `not_attempted`). This catches a custom harness that
107> preserved `active_snapshot` across the Stage 15 pass increment without
108> re-pinning. The reference meta-agent re-pins every pass, so this check never
109> fires there. Block B itself cannot detect this (it is `snapshot_id`-only, not
110> `pass`-aware).
111
112Notes: `workspace/findings/`, `workspace/archive/`, `workspace/.tx_log.jsonl`,
113`workspace/helpers/` and `.mantis_state.json` are STATE-RELATIVE (under
114--state_root). Any code snippet you inspect for a finding is SNAPSHOT-RELATIVE
115(under CODE_ROOT). Never write under CODE_ROOT.
116
117Review a list of security findings and merge duplicate findings that refer to
118the exact same security flaw or adjacent code paths.
119
120Execute your task as follows:
121
1221. **Load Raw Findings & Archived Findings Queue:**
123
124 - List the contents of the directory and read the files in
125 `workspace/findings/`. If the directory is empty or does not exist, notify
126 the user and exit.
127 - *Important:* Ignore hidden files and directories (such as the `.trash/`
128 subdirectory) when listing or processing findings.
129 - Locate and load all archived finding JSON files from previous loop passes,
130 if they exist, under `workspace/archive/findings_pass_*/*.json` and
131 `workspace/archive/loop*_findings/*.json`. These files represent
132 vulnerabilities that have already been fully evaluated, triaged, and
133 potentially patched in previous passes.
134 - *Important:* Do NOT read or deduplicate against
135 `workspace/historical_learnings.jsonl` (VCS history), as we want to catch
136 regressions if old bugs were reintroduced.
137
1382. **Filter Loop Duplicates (snapshot-gated).** First, stamp `discovery_commit`
139 on any CURRENT finding that lacks it, using `active_snapshot.snapshot_id`
140 from `.mantis_state.json` (skip when unpinned). Then, for each current
141 finding that matches an archived finding (by `code_paths`+`title`
142 similarity), run:
143
144 **Signature-based candidate matching (Phase 3) — TIGHTENS, never replaces:**
145 `signature` may only PROMOTE a pair to "candidate for the pairwise snapshot
146 check"; it may NEVER by itself cause a hard DUPLICATE/trash. A pair is a
147 candidate for the Pairwise Snapshot Match Check below ONLY if it satisfies
148 BOTH:
149
150 - it matches under today's `code_paths` + `title` similarity, comparing
151 `code_paths` entries line-inclusively (WITH their trailing `:line`); AND
152 - (when both findings have a `signature`) their `signature` fields are equal.
153 A `signature` match WITHOUT the `code_paths` + `title` agreement is NOT a
154 duplicate — at most a soft `possible_duplicate_of` (keep the finding
155 ACTIVE), never a trash. Rationale: `signature` strips the line number and
156 all-but-first `code_paths` entry, so two DISTINCT bugs in the same file
157 (e.g. `parser.c:100` vs `parser.c:900`) with the same title+CWE share one
158 `signature`; trashing on signature alone would silently delete a real
159 finding. If EITHER lacks `signature`, use today's `code_paths` + `title`
160 similarity matching unchanged.
161
162 PAIRWISE SNAPSHOT MATCH CHECK (decides MATCHED vs NOT_MATCHED) — compares the
163 CURRENT finding's `discovery_commit` against the ARCHIVED finding's
164 `discovery_commit` for this pair (NOT against `SNAPSHOT_ID`):
165
166 1. If `snapshot_pinned` is false AND there is NO `active_snapshot` in state
167 (MODE-OFF) -> NOT_MATCHED. Stop. (In HALT — `active_snapshot` present but
168 `snapshot_pinned=false` — do NOT short-circuit here; fall through to the
169 pairwise comparison below, which will be NOT_MATCHED because the current
170 finding's `discovery_commit` is a `live:` id that will not equal the
171 archived one.)
172 2. Read the CURRENT finding's `discovery_commit` and the ARCHIVED finding's
173 `discovery_commit`:
174 - If EITHER is missing, empty, or the literal `"MIXED"` -> NOT_MATCHED.
175 - If they are NOT byte-for-byte equal to each other -> NOT_MATCHED.
176 - If they ARE byte-for-byte equal to each other (both present, non-MIXED)
177 -> MATCHED. There is no other route to MATCHED; never fuzzy-compare. The
178 global "default the field and proceed" backward-compat rule does NOT
179 apply to `discovery_commit`: absent = NOT_MATCHED. (There is NO separate
180 "dirty" gate: a dirty tree's SNAPSHOT_ID already embeds the working-tree
181 content hash, so within-pass findings MATCH and cross-pass bare-commit
182 findings do not.) Note: this is a PAIRWISE check (current vs archived),
183 NOT a check against the global `SNAPSHOT_ID` — dedupe stamps the current
184 finding's `discovery_commit` to `SNAPSHOT_ID` in Step 2 above, so a
185 check against `SNAPSHOT_ID` would always be MATCHED and would trash
186 reintroduced/regression bugs as DUPLICATE.
187
188 Then, using the idempotency rule (Input/Output Contract → Idempotency
189 Guarantee) to avoid double-writes, decide mechanically:
190
191 - **MATCHED** (both present and equal): soft-delete the current finding as a
192 loop-duplicate exactly as before — set `"status": "DUPLICATE"` and
193 `"duplicate_of": "<archived_uuid>"`, clear `possible_duplicate_of` if
194 present, ensure `mkdir -p workspace/findings/.trash/`, move it there, and
195 log a `loop_filter` transaction in `workspace/.tx_log.jsonl`. If the
196 current finding lacks `lineage_id` but the archived finding has one,
197 inherit the archived finding's `lineage_id` onto the current finding before
198 moving it (so the lineage chain is preserved across the merge).
199 - **NOT_MATCHED** (differ, or either absent): do NOT set `DUPLICATE` and do
200 NOT move to trash. Keep the current finding ACTIVE and set
201 `"possible_duplicate_of": "<archived_uuid>"` (a soft, non-terminal hint).
202 If the current finding lacks `lineage_id` but the archived finding has one,
203 inherit the archived finding's `lineage_id` onto the current finding (so
204 the lineage chain is preserved for report folding even when the findings
205 are on different snapshots).
206
207 > [!IMPORTANT] **STATUS & DUPLICATE INVARIANTS:**
208 >
209 > - A finding MUST NOT carry both `duplicate_of` and `possible_duplicate_of`
210 > pointing to the same target UUID.
211 > - `status = "DUPLICATE"` MUST NOT coexist with `possible_duplicate_of`
212 > pointing to the same target UUID.
213 > - Under **NOT_MATCHED** (outside the MODE-OFF fallback exception), the
214 > finding's `status` MUST remain active (e.g. `VALID`,
215 > `PROVISIONALLY_VALID`, `NEEDS_RESEARCH`), `duplicate_of` MUST NOT be set,
216 > and the finding MUST NOT be moved to `.trash/`. Setting
217 > `possible_duplicate_of` is a non-terminal hint only.
218
219 - **POSSIBLE REGRESSION:** if the archived match has a RESOLVED status
220 (`patch_status` in {`VERIFIED_SECURE`,`MITIGATION_PROPOSED`} OR
221 `status`==`FALSE_POSITIVE` OR `production_viability`==`NON_VIABLE`) AND the
222 pair is NOT MATCHED, keep the current finding ACTIVE, add a history note
223 "POSSIBLE REGRESSION vs \<archived_uuid>", and do NOT filter it. (A
224 reverted fix re-discovered on new code must never be trashed.)
225 - **EXACT-UUID retry exception (unchanged):** if the current finding has the
226 EXACT SAME UUID as the archived one, it was intentionally copied back for a
227 retry — do NOT filter it (keep as-is).
228 - **Permanently-unpinned exception (MODE-OFF only — no `active_snapshot`):**
229 if there is NO `active_snapshot` in state (MODE-OFF = today's default; a
230 target with no snapshot boundary, e.g. a live endpoint), `snapshot_pinned`
231 is false at the PASS level (`active_snapshot.snapshot_pinned` — it is NOT a
232 per-finding field) and Block B is uninformative — fall back to today's
233 dedup by `signature` if present, else `stable_key` = normalized_title +
234 first `code_paths` entry **including its trailing `:line`**
235 (line-inclusive, same as Step 2). Rationale: stripping `:line` would
236 collapse two DISTINCT bugs in the same file (e.g. `parser.c:100` vs
237 `parser.c:900`) with the same title+CWE into one — silently deleting a real
238 finding. Note: `signature` itself strips `:line` by design (it is a coarse
239 identity for cross-pass lineage, not a dedup key); this fallback therefore
240 prefers `signature` only when `stable_key`'s line-inclusive match ALSO
241 agrees, never on `signature` alone. This preserves dedup for targets that
242 can never MATCH. When `active_snapshot` IS present but unpinned (HALT
243 mode), this exception does NOT fire: keep the snapshot-gated behavior above
244 (NOT_MATCHED → keep ACTIVE + `possible_duplicate_of`, never `DUPLICATE`).
245 **POSSIBLE REGRESSION takes precedence over this fallback:** a
246 resolved-archived finding paired with a NOT_MATCHED current finding is
247 ALWAYS kept active (never trashed), regardless of mode.
248
2493. **3-Tier Deduplication Ladder (Deterministic & Semantic Matching):** When
250 resolving finding lineages and evaluating deduplication candidates, the
251 deduplicator uses a hierarchical 3-tier ladder designed for sub-millisecond
252 fast-path execution with semantic RAG fallback:
253
254 - **Tier 1 (Fast-Path Exact Heuristic Anchors — < 1ms, 0 tokens):**
255
256 1. Exact content identity signature match:
257 `hashlib.sha256(canonical_fp | canonical_cwe | target_symbol)`
258 (invariant to line shifts, backticks, and title paraphrasing).
259 2. Exact `canonical_filepath + normalized CWE + target_symbol` match.
260 3. Strict line proximity window ($\\le 3$ lines) on exact canonical
261 filepath when target symbol is empty.
262
263 - *Result*: If matched, immediately inherit ancestor `lineage_id` without
264 LLM or embedding overhead.
265
266 - **Tier 2 (RCA Normalization):** If Tier 1 heuristic matching does not
267 produce an exact anchor, synthesize a standardized 5-line Root Cause
268 Analysis (RCA) summary:
269
270 - `Component`: Normalized canonical filepath and symbol.
271 - `Vulnerability Class`: Canonical CWE taxonomy identifier and name.
272 - `Root Cause Mechanism`: Underlying programming or logic defect.
273 - `Failure Condition`: Specific input, state, or boundary condition.
274 - `Taint Dataflow`: Source-to-sink dataflow trajectory.
275
276 - **Tier 3 (Vector Embedding & Cosine Similarity Scan):**
277
278 - Project the standardized RCA summary into dense vector embeddings using
279 the configured multi-provider embedding engine (default:
280 `vertex_ai/gemini-embedding-001`).
281 - Perform nearest-neighbor scan over historical `lineage_vectors` in the
282 database using bounded cosine similarity.
283 - **CWE Family & Class Structural Guard:** When comparing query finding
284 vectors against candidate lineage records, if both findings have explicit
285 CWE classifications (e.g. CWE-89 vs CWE-78), normalize them. If they
286 belong to distinct, incompatible CWE IDs, skip candidate vector
287 comparison entirely to structurally prevent false-merging distinct
288 vulnerability classes regardless of cosine score.
289 - **Positive threshold ($\\ge 0.90$):** Semantically equivalent findings
290 with differing phrasing, scanner labels, or line shifts merge into the
291 same ancestor `lineage_id` (default: 0.90, configurable via
292 `EMBEDDING_SIMILARITY_THRESHOLD`).
293 - **Negative threshold ($< 0.70$):** Distinct vulnerability classes (e.g.,
294 SQLi vs Command Injection, Stored vs Reflected XSS) maintain low
295 similarity and fail closed, minting a fresh unique `lineage_id`.
296
2974. **Filter Duplicate Findings in Current Batch:** Check the current findings
298 against each other to find duplicates. Two findings are duplicates ONLY if
299 they share the same `code_paths` entry **line-inclusively** (WITH trailing
300 `:line`) AND have the same or highly similar title. If multiple findings
301 refer to the exact same flaw at the same location, they must be merged.
302 Findings at different lines in the same file are DISTINCT — never merge them.
303
3045. **Map/Reduce Chunking Strategy (For Scale):** If there are many finding files
305 (e.g., > 20 items), use a Map/Reduce approach to group them by target file or
306 component before checking for overlaps to avoid context window limits.
307
3086. **Token-Optimized Consolidation and Merging:** To minimize LLM output tokens
309 and prevent data loss, **do not manually rewrite or output the merged JSON
310 files in your response.** Instead, follow this pattern:
311
312 1. **Identify Duplicates:** Internally map which findings are duplicates of a
313 primary finding.
314 2. **Reusable Deterministic Scripting (versioned):** Write a reusable helper
315 script (e.g. `workspace/helpers/merge_findings.py`) whose FIRST LINE is
316 exactly `# MANTIS_HELPER_VERSION = 2`. Before reusing an existing helper,
317 grep its first lines for `MANTIS_HELPER_VERSION = 2`; if that marker is
318 absent or a different integer (a helper left by an older pipeline
319 version), REGENERATE the helper. Only reuse it when the marker matches.
320 The script must follow these deterministic rules:
321 - **Title:** Pick the most comprehensive and descriptive title.
322 - **ID:** Preserve the unique `"id"` of the primary finding being kept.
323 - **Severity:** Pick the highest severity level specified among the merged
324 items.
325 - **Privileges Required:** Inherit the most severe privilege requirement
326 (priority: `NONE` > `LOW` > `HIGH`).
327 - **Attacker Position:** Inherit the most critical position requirement
328 (priority: `EXTERNAL` > `INTERNAL_NETWORK` > `IN_CLUSTER` > `LOCAL` >
329 `HOST_SYSTEM` > `SUPPLY_CHAIN` > `PHYSICAL_TEMPORARY` >
330 `PHYSICAL_LONG_TERM`).
331 - **User Interaction:** Inherit the most severe user interaction
332 requirement (priority: `NONE` > `REQUIRED`).
333 - **Code Paths:** Collect and deduplicate all file paths and line numbers
334 into a single unique array.
335 - **Description, Mitigation, & Impact:** Concatenate cleanly.
336 - **History:** Concatenate and preserve all `"history"` entries from the
337 merged findings. Append a new entry to the `"history"` array for this
338 merge action conforming to the schema (containing `"stage": "dedupe"`,
339 `"action": "merge"`,
340 `"details": "Merged duplicate findings: [comma-separated-ids]"`,
341 `"pass_number": <current_pass_number>`, and
342 `"timestamp": "<current_iso8601_timestamp>"`).
343 - **Unknown keys & provenance (MANDATORY):** The script MUST copy through
344 EVERY key it does not explicitly handle (including `discovery_commit`,
345 `repro_snapshot_id`, `patch_base_snapshot`, `possible_duplicate_of`,
346 `signature`, `lineage_id`, `cwe`) from the primary finding onto the
347 merged object — never drop unknown fields. It MUST REFUSE to merge two
348 findings whose `discovery_commit` values differ (they describe different
349 code versions); leave them separate and log the refusal. When merging
350 findings that all share one `discovery_commit`, preserve it unchanged.
351 3. **Execute the Script:** Run your script to update the primary finding's
352 file (`workspace/findings/<primary_id>.json`) on disk.
353
3547. **Transactional Staged Clean Up:** Do not permanently delete redundant files.
355 Ensure the trash directory exists (e.g.,
356 `mkdir -p workspace/findings/.trash/`). Before moving, the script must update
357 the duplicate finding files, setting `"status": "DUPLICATE"` and
358 `"duplicate_of": "<primary_uuid>"`. Move the merged duplicate `.json` files
359 to the trash staging directory (`workspace/findings/.trash/`). For every file
360 moved, append a transaction record to `workspace/.tx_log.jsonl`.
361
362 ### Transaction Log Schema Format (`workspace/.tx_log.jsonl`)
363
364 Each line must be a self-contained JSON object documenting the transaction:
365
366 ```json
367 {"timestamp": "2026-07-14T15:13:00Z", "action": "loop_filter | dedupe_merge", "primary_uuid": "[UUID] (or null for loop_filter)", "moved_uuid": "[UUID]"}
368 ```
369
370 This cleans up the directory for downstream stages while preserving rollback
371 capability.
372
373When complete, notify the user.