# Validate Findings > Use when the user asks to validate, reproduce, exploit, red-team, or live-test security findings against a Kubernetes/OpenShift cluster, operator, WASM module, container, pod, or cluster component. Ingests *-security-audit, *-threat-model, and *-triage reports, builds an attack plan (replay + chained + novel), executes it against an authorized live target under a hard scope guard, and emits a schema-validated *-validation.{json,md} report. - Skill: `openshift/validate-findings` (Agent Skill, multi-file: 28 files) - Install (CLI): `npx skillmds@latest add openshift/validate-findings` - Raw SKILL.md: https://api.skillmd.com/api/skills/openshift/validate-findings/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Web & Frontend - Author: openshift (https://skillmd.com/u/openshift) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/openshift/validate-findings --- # Validate Findings — Live Validation & Attack-Chain Harness > **Paths.** `analysis-results/…` and `progress-tracker/…` in this skill are the > default workspace layout. They resolve through `locations.yaml` in > `$TRAUST_CONFIG_HOME` (`docs/setup.md`, Storage locations); substitute your > configured roots. Take the static outputs of `secure-code-audit`, `threat-model`, and `triage` and **prove or refute them against a live authorized environment**. Then go further: chain confirmed findings into multi-step kill-chains and hunt for novel attacks the static analysis missed. > **Authorization**: This skill executes potentially state-changing actions against live infrastructure. It MUST only be run against targets you own or have explicit written authorization to test. Every action is gated by the scope guard (`scope.py`) — out-of-scope steps are refused and logged, never silently skipped. --- ## Input `$ARGUMENTS` is parsed as whitespace-separated tokens. The **first non-flag token** is the findings source; remaining tokens are flags. ### Findings source (required — one of) | Form | Resolution | |---|---| | `/` | Directory containing `*-security-audit.{json,md}`, `*-threat-model.md`, `*-triage.{json,md}` | | `.{md,json}` | A single report file; siblings auto-discovered in the same directory | | `/` | Shorthand resolved to `../analysis-results/findings///` | | `-findings` | Product package from `../progress-tracker/processed-results/-findings/`. All repos in the package are ingested into a single model; each finding is tagged with its source repo. The bare `` form (without `-findings`) is also accepted. | | `,,…` | **Multiple packages** merged into one model — used when a logical product spans several `processed-results` entries (see the Deduplication Map in `progress-tracker/VALIDATION-PRIORITY-LIST.md`, e.g. ODF → 12 packages). Findings are tagged `:` and de-duplicated by `(repo, id)` so a repo appearing in two packages contributes once. | | `all-confirmed` | Every triage-confirmed finding across `../analysis-results/findings/**` (use with `--dry-run` first) | **Refuted findings stay in scope.** When a `-refuted-register.json` sits next to the audit report (emitted by python3 -m traust.cli ledger emit-triage), its entries are candidate targets, not exclusions: a finding dismissed as a false positive — even human-countersigned — is a falsifiable claim, and a reproducing exploit from this skill overrides the assertion via evidence-class precedence (`fp_overridden`, loudly surfaced with the countersigner attributed). Include register entries in attack planning when scope and budget allow. **Impact-analysis artifacts as target selection.** When `analysis-results/impact/-impact-analysis.json` (from `/impact-analysis`) covers the finding's CVE, use it two ways: (a) prioritization — repos classified `affected` with triage's `attacker_influence: plausible` are the validation candidates worth cluster time first; (b) attack planning — the repo's `evidence.govulncheck_trace` names the call path caller-first, so the first in-repo frame identifies which interface the attack plan should drive. A confirmed exploitation here is execution evidence (class 1): it flips the finding to `confirmed` in the disposition ledger and is the terminal answer to "is this repo actually affected". **Product packages** (`*-findings/` directories) contain sub-directories grouped by repo or by sub-group/repo (1–2 levels). The harness auto-detects the package layout: if the source directory has no audit reports at the top level but has sub-directories with audit reports, it is treated as a package and all repos are ingested. Findings from each repo carry a `source_repo` tag for traceability in the plan, execution log, and validation report. ### Scope binding (at least one mode; modes are additive) | Flag | Mode | Effect | |---|---|---| | `--targets ` | **1 — explicit** | Load a rules-of-engagement scope file (see `targets.example.yaml`). Highest precedence. | | `--context ` | **2 — inline** | Kubeconfig context to bind (repeatable). Implies the cluster at that context is in scope. | | `--ns ` | 2 | Namespace allowlist (repeatable, glob OK). | | `--image ` | 2 | Container image allowlist (repeatable, glob OK). | | `--pod ` | 2 | Pod label selector allowlist (repeatable). | | `--container ` | 2 | Running container name/ID for the container adapter (repeatable, glob OK). | | `--wasm ` | 2 | WASM artifact path for the wasm adapter (repeatable). | | `--infer-scope` | **3 — inferred** | Derive scope from report `metadata` + threat-model `entry_points`. Lowest precedence; **never** infers `kube-system`, `openshift-*` control-plane, or `default` namespaces. | If no scope mode is given, the harness runs `--dry-run` implicitly and warns. ### Execution control | Flag | Effect | |---|---| | `--dry-run` | Stop after Phase 2. Emit `attack-plan.yaml` only; nothing touches the target. | | `--auto` | Skip the Phase 3 review gate. Use **only** in isolated lab environments. | | `--replay-only` | Validate existing findings only; skip chaining and novel hunting. | | `--novel-only` | Skip replay; run recon + chain synthesis + novel probes only. | | `--destructive` | Permit steps the adapter classifies as `destructive` (data loss, DoS, irreversible mutation). Without this flag such steps are recorded as `not_attempted` with reason `destructive-not-permitted`. | | `--max-novel ` | Cap novel-attack hypotheses (default 10). | | `--out ` | Override output directory (default: alongside the source reports). | --- ## Scope Binding — Resolution & Enforcement Run `python harnessing/5-validate/validate-findings/scope.py` semantics: 1. **Load** mode-1 file if `--targets` given; merge mode-2 inline flags on top; if `--infer-scope`, call `ingest.infer_scope()` and merge with **lowest** precedence. 2. **Compile** into a `Scope` object exposing `is_in_scope(action: Action) -> (bool, reason)` where `Action = {adapter, verb, context, namespace, resource, name, image, extra}`. 3. **Hard denies** (always refused regardless of flags or `--auto --destructive`): - Any entry in `targets.yaml#off_limits`. - Any namespace matching `kube-system`, `openshift-etcd`, `openshift-kube-apiserver*`, `openshift-authentication*` **unless explicitly listed** in mode-1 `clusters[].namespaces`. - `expires` date in the past. 4. **Preflight** every bound target via the adapter's `preflight()` to capture a fingerprint (cluster version, node count, image digest, WASM sha256). The fingerprint goes into `validation.json#metadata.target_fingerprint` so results are reproducible. Every executed step is logged with the scope-check outcome. A `blocked_by_scope` verdict is a **result**, not an error — it tells the reader the PoC would have crossed a boundary the engagement does not permit. --- ## Benchmark mode (P7) `analysis-results/scan-testing/validation-benchmark/benchmark-findings.json` (from python3 -m traust.cli sweep benchmark plan) is a first-class input: validate its claims against the deployed fixtures exactly like campaign findings, then score the report with `run_validation_benchmark.py score --variant vuln|safe`. Benchmark runs live in the harness-QA tree and never touch campaign metrics. ## Phase 0b — Target attestation (fail-closed, P2) After scope loads and before ANY probe executes, attest the target and write `target-attestation.json` into the validation output dir: ```bash python3 -m traust.cli admin attest-target --out /target-attestation.json \ --namespace [--csv ] [--selector ] \ [--version --affected-range '] [--context ] ``` Record the summary in `metadata.target_attestation`. If `attested: false`: probes MAY still run for diagnosis, but every verdict in the report is structurally void — `emit_validation_ledger_events` routes the entire run to `needs_review` as `environment_invalid` (or `attestation_missing` for post-0.176.0 reports without the artifact). The rhoso.v2 failure class — 34 false "refuted" from a never-installed operator — becomes 0 verdicts at the front door. ## Phase 0c — Positive-control pairing (assay validity, P1) Every **refutation-capable** probe step carries paired positive controls in its plan entry, executed in the same session and recorded in the step's `controls[]` (schema `positive_control`): | Probe class | Paired control (kind) | |---|---| | RBAC "subject X cannot do Y" | same client performs an action X is KNOWN to be allowed (`must_succeed`) — proves auth worked, enumeration non-empty, API reachable | | Secret-exposure "secret not readable" | read a planted canary secret the probe SHOULD see (`must_succeed`) — proves the oracle observes the right store | | Network/exposure "endpoint not reachable" | reach a known-open endpoint on the same path (`must_succeed`) and a known-closed one (`must_deny`) | Rules (enforced by the soundness gate + ledger emitter): - A `refuted` verdict with a **failed** control is quarantined as `failed-positive-control` — any report age: the assay demonstrably didn't work. - A `refuted` verdict with **no passing control**, on reports at/after harness 0.177.0, quarantines as `missing-positive-control`. - Controls never substitute for the probe: they prove the assay could have detected the claim, converting "absence of evidence" into "evidence of absence, with the assay proven live." ## Phase 0d — Differential probing for authz claims (P3) For every "role/subject can(not) do X" refutation, probe the **pair** in the same session and record it in the step's `differential` block: - the claimed action, AND - a neighbor action with a **known-different expected outcome** (same subject → a verb it IS allowed; same verb → a subject that IS authorized). If both outcomes are identical, the oracle cannot discriminate allowed from denied — the refutation is unsound **regardless of which way it pointed** (`non-discriminating-oracle`, quarantined at any report age). Post-0.178.0, an authz-class refutation without a discriminating differential quarantines as `missing-differential-probe`. One extra request per probe; catches wrong-oracle failures without knowing *why* the oracle is wrong. ## Phase 0e — Severity validation on confirmations (P9) Every `confirmed` finding records `severity_validation`: the CVSS components the exploit ACTUALLY demonstrated (attack vector used, privileges the probe identity held at success, user interaction, scope crossing observed from the E0 artifact, impact axes evidenced) and the signed delta vs the claimed score. When |delta| >= 1.0 **and** the evidence grade is E0/E1/E2 (never E3), emit a `proposal` — the emitter routes it to the countersign severity decision as a needs_review item; machines never write `disposition.severity`. A demonstrated DOWNGRADE ("exploitation required cluster-admin") is exactly as valuable as an upgrade. Findings that are constituents of a demonstrated attack chain also record `chain_context` (chain id, chain severity, role) — chain membership is severity evidence and belongs in the proposal rationale. ## Phase 6 pointer — discovery sweeps & replay artifacts (P5/P8) **Replay (P8, required on every probe):** record a self-contained replay script per executed step in `artifacts/replay/.sh` (+ inputs file when the probe posts data) and reference it in the step's `replay` block with the attestation fingerprint sha. A countersign human re-runs the exact probe instead of trusting the transcript. **Discovery sweeps (P5, optional phase after replay/chained/novel):** outputs are NEW finding candidates with `origin: validation-discovery`, routed to `/triage` generic ingest — never straight to the ledger: 1. *State diffing (implemented):* python3 -m traust.cli impact cluster-state-diff snapshot before the sweep and after it; `diff` turns unexpected deltas (new/mutated RBAC, SCC changes, webhook mutations, deleted NetworkPolicies, new exposure) into candidates, excluding declared probe side-effects (`--expected`). 2. *Anonymous-surface sweep* — unauthenticated reachability vs an authenticated baseline (per-sweep increment, not yet implemented). 3. *Privilege-escalation chain search* — seeded from operator-priv-profile + the RBAC graph; attempt the cheapest link under the scope guard (not yet implemented). 4. *Browser role×route matrix + IDOR probing* (browser lane; not yet implemented). All sweeps inherit the hard scope guard unchanged. ## Phase 1 — Ingest & Normalize ```bash python harnessing/5-validate/validate-findings/ingest.py > /tmp/normalized.json ``` 1. Locate the three report types in the source directory. JSON is preferred; fall back to Markdown parsing when JSON is absent. 2. Build a unified `Finding` list. For each finding merge: - `security-audit`: `id`, `title`, `severity`, `cwes`, `locations`, `evidence[]`, `attack_pattern`, `cvss` - `triage` (if present): `verdict`, `verify_verdict`, `confidence`, `preconditions[]`, `first_links[]`, `severity_label`, `owner_hint` - `threat-model` (if present): linked `threat_id`, `entry_point`, `asset`, `controls` 3. Build a `ThreatGraph` skeleton: assets (with sensitivity), entry points (with trust level + reachable assets), threats (with status + linked finding evidence). 4. Extract **embedded PoCs**: scan `evidence[]` code blocks and `attack_pattern` text for fenced `yaml`/`bash`/`json`/`curl` blocks and the triage `rationale` STEP narratives. Tag each as `{lang, body, source_field}`. Findings with triage `verdict == false_positive` are carried through but default to `technique: skip` in the plan (override with `--include-fp`). Findings with triage `verdict == needs_review` (evidence not statically locatable; never proven false) are **not** skipped — they are planned like `true_positive` and the resulting step is tagged `triage_was_needs_review: true`. --- ## Phase 2 — Attack Planning ```bash python harnessing/5-validate/validate-findings/plan.py /tmp/normalized.json --scope > /-attack-plan.yaml ``` The plan is an ordered list of **steps**, each: ```yaml - id: step-007 finding_ref: ODR-2026-004 # or novel_ref / chain_ref technique: replay # replay | adapted | chained | novel | recon adapter: k8s verb: apply-manifest target: {context: lab-spoke-1, namespace: ramen-ops} payload: | # the literal manifest / command / wasm-invoke apiVersion: ramen.openshift.io/v1alpha1 kind: Recipe ... classification: mutating # safe | mutating | destructive (from adapter.classify) expected: "pods/exec succeeds in namespace outside tenant scope" rollback: "kubectl --context lab-spoke-1 -n ramen-ops delete recipe pwn" preconditions: [step-003] # steps that must succeed first ``` ### 2a. Replay steps For each non-FP finding, convert extracted PoCs into one or more steps: | PoC form | Adapter / verb | |---|---| | YAML manifest (Kind present) | `k8s / apply-manifest` | | `kubectl …` / `oc …` literal | `k8s / raw` | | `curl` / `http` against a Service | `k8s / port-forward+http` (or `container / network-probe` if no cluster) | | `podman exec` / `docker exec` | `container / exec` | | Shell against a pod | `k8s / exec` | | WASM invoke / hostcall | `wasm / invoke-export` | | No literal PoC | `technique: adapted` — synthesize a minimal probe from `cwes` + `locations` (e.g. CWE-918 ⇒ SSRF callback to a harness-controlled listener) | ### 2b. Chained steps (`chain.py`) Build the attack graph and emit one `chain_ref` step-group per discovered path: - **Nodes**: findings, threat-model entry points, threat-model assets, recon-discovered resources (added in Phase 4 second pass). - **Edges**: - `entry_point → finding` when the finding's `preconditions` are satisfiable at that entry point's trust level. - `finding_A → finding_B` when A's expected impact (token theft, ns escape, RBAC grant, file write, network position) satisfies any of B's preconditions. Use the capability vocabulary in `chain.py:CAPABILITIES`. - `finding → asset` when the finding's impact references the asset. - **Search**: DFS from every entry point to every asset with sensitivity ≥ high. Deduplicate by node-set. Cap path length at 6. - For each chain, emit ordered steps reusing the replay steps where they exist, inserting `technique: chained` glue steps (e.g. "extract SA token from step-003 evidence and set KUBECONFIG for step-004"). - Annotate with **MITRE ATT&CK** technique IDs from the shared pinned tables (`harnessing/attack-coverage/tables/`, loaded via `attack_refs.capability_map()`; IDs validated against the pinned ATT&CK release at import). ### 2c. Novel steps (`novel.py`) 1. **Recon plan**: emit `technique: recon` steps for the bound target type(s) — these are always `classification: safe`: - **k8s/operator/component**: `kubectl get crd,clusterrolebinding,rolebinding,networkpolicy,validatingwebhookconfiguration,mutatingwebhookconfiguration,svc,ep -A -o json`; `kubectl auth can-i --list --as=`; `kubectl get pods -o jsonpath` for securityContext, hostPath, SA mounts, capabilities. - **container**: `inspect`; `cat /proc/self/status` (CapEff), `/proc/mounts`, env, listening sockets. - **wasm**: `wasm-tools dump --imports --exports`; component-model world inspection. 2. **Diff** recon output against threat-model `entry_points` + `controls`. Surfaces present live but absent from the model are **candidate novel entry points**. 3. **Crypto posture probes** — when findings include `category: cryptography` or `pqc_classification` and a cluster context is bound, run runtime data collection: ```bash python3 -m traust.cli adapters crypto-audit cluster --context --namespaces [--discover] [--groups "..."] ``` This emits `crypto-audit/v1` (`tier: runtime`) with raw TLS negotiation data, crypto-policy state, and OpenSSL/Go versions. For source and image tier crypto analysis, delegate to the `crypto-analysis` skill. These facts feed the PQC PROBE_CATALOGUE entries and `chain.pqc_caps_from_probe()` for capability mapping. Record each run in `metadata.tools` and `deterministic_steps`. 4. **Pattern probes** — for each candidate, emit a probe step from the catalogue in `novel.py:PROBE_CATALOGUE` keyed by target type and surface class: - *Operator*: confused-deputy (cross-ns `*Ref` fields in CRD schema), SSRF (URL-typed CRD fields → harness listener), TOCTOU (status-subresource race), webhook bypass (label/annotation that skips admission), over-broad RBAC (`auth can-i --list` shows `* * *`). - *Pod / container*: writable hostPath, `CAP_SYS_ADMIN`/`CAP_NET_ADMIN` presence, `/var/run/docker.sock` or CRI socket mount, SA token with cluster-admin reachability, procfs breakout, shared PID/net namespace. - *Cluster component*: unauth metrics/pprof/healthz with side-effects, anonymous-auth on kubelet, insecure-port residue, etcd client cert reuse. - *WASM*: unrestricted WASI preopen dirs, hostcall import without capability check, linear-memory OOB on exported function, missing fuel/epoch limits. 5. **AI-assisted hypotheses** — after recon executes (Phase 4 first pass), reason over: recon output, unexploited threat-model threats (`status: unmitigated` with no linked finding), and the CWE distribution of confirmed findings. Propose up to `--max-novel` additional attack hypotheses. Each becomes a `technique: novel` step with a clear `expected` outcome that distinguishes confirmed from refuted. ### 2d. Ordering & classification Order: all `recon` → all `replay` (severity desc) → `chained` → `novel`. Run `adapter.classify()` on every step; if `destructive` and `--destructive` not set, mark `skip: destructive-not-permitted`. Write the plan to `/-attack-plan.yaml`. --- ## Phase 3 — Review Gate Unless `--auto` or `--dry-run`: 1. Print a summary table: step count by `{technique × classification}`, every `mutating`/`destructive` step's verb + target, and every step that `scope.is_in_scope()` would refuse. 2. Ask the user to approve. Accept: `yes` (run all permitted), `edit` (open `attack-plan.yaml`, re-read, re-summarize), `skip ` (mark listed steps `not_attempted: user-skip`), or `no` (exit, plan kept on disk). 3. Record the approval (who, when, mode) in `validation.json#metadata.approval`. If `--dry-run`: stop here. The plan on disk is the deliverable. --- ## Phase 4 — Execute ```bash python harnessing/5-validate/validate-findings/execute.py /-attack-plan.yaml --scope --out ``` For each step in plan order: 1. `scope.is_in_scope(step.action)` — if false, record `verdict: blocked_by_scope` with reason; continue. 2. Check `preconditions` — if any referenced step did not yield `verdict: confirmed`, record `verdict: not_attempted` with reason `precondition-failed:`; continue. 3. Dispatch to `TARGET_ADAPTERS[step.adapter].execute(step, scope, audit)`: - Capture stdout/stderr/exit, plus adapter-specific evidence (resource YAML before/after, HTTP response body, WASM trap). - Write artifacts to `/artifacts/step-.*` and compute sha256. - Compare observed outcome to `step.expected` → `verdict ∈ {confirmed, refuted, inconclusive}`. - **Refutation-soundness gate** (`soundness.py`, applied deterministically by `execute.py` — never bypass it when composing verdicts by hand): `refuted` is **un-emittable** when (a) the probe transcript matches an error signature (`Forbidden`/validator-RBAC denial, `command not found`, jsonpath errors, `NotFound`, `does not exist`, `Could not connect`, an unsubstituted `{ns}`/`{sa}` template), (b) an RBAC-style probe enumerated zero concrete subjects (empty `vf-rbac-tried:` list) or carries a literal `system:serviceaccount:{ns}:{sa}` placeholder, or (c) the run directory contains `install-failure.yaml` (whole run — the operand never deployed). Gated verdicts downgrade to `inconclusive` and carry a machine-readable `soundness_flag` (`error-signature:` | `rbac-zero-subjects` | `rbac-template-placeholder` | `target-not-deployed`). Downstream, flagged findings route to `needs_review` — never to the false-positive/countersign path (`emit_validation_ledger_events.py`). 4. Append one JSONL line to `/validation-audit.jsonl`: ```json {"ts":"","step_id":"...","adapter":"...","verb":"...","target":"...","scope_check":"pass","classification":"...","verdict":"...","evidence":["artifacts/step-007.log"],"rollback":"..."} ``` 5. If `classification: mutating` and the adapter provides `rollback()`, run it immediately after evidence capture and record `rollback_performed: true|false` with output. 6. **Two-pass for novel**: after recon steps complete, re-invoke `novel.py` with live recon artifacts to materialize the AI-assisted hypotheses (Phase 2c-4) into concrete steps, append them to the plan, and continue execution. Rate-limit: ≤ 1 mutating step per 2 s per cluster context; back off ×2 on 429/503 from the API server. --- ## Phase 5 — Report ```bash python harnessing/5-validate/validate-findings/report.py > /-validation.json python3 -m traust.cli reporting validate --schema contracts/schemas/validation.schema.json /-validation.json python harnessing/5-validate/validate-findings/report.py --markdown > /-validation.md ``` `-validation.json` conforms to `contracts/schemas/validation.schema.json`: - `metadata`: `date`, `harness_version` (`VERSION` + short SHA), `scope_binding_mode` (`explicit|inline|inferred|mixed`), `target_fingerprint[]`, `approval`, `flags`. - `source_reports[]`: paths + sha256 of the ingested audit/threat-model/triage files. - `summary`: counts by verdict, by technique, novel-finding count, highest-impact confirmed chain. - `validated_findings[]`: one per source finding — `source_id`, `verdict`, `technique`, `evidence[]`, `observed_impact`, `deviation_from_claim`, `rollback_performed`, plus `soundness_flag` when the refutation-soundness gate downgraded a `refuted` to `inconclusive` (see Phase 4; `report.py` re-applies the gate as a backstop at rollup). - `attack_chains[]`: `chain_id`, `name`, `entry_point`, `terminal_asset`, `mitre_attack_refs[]`, `steps[]`, `verdict`. - `novel_findings[]`: full `report.schema.json#/$defs/finding` objects + `discovery_method` + `chain_context` — ready to feed into `file-security-defect`. **Routing novel findings (user directive 2026-07-27 — direct ledger entry, no `/triage` precondition).** A novel finding discovered by live validation is a first-class campaign finding at birth, same convention as `/vuln-scan` supplements and `/verify-remediation` regression routing (python3 -m traust.cli route regressions is the reference mechanics): mint its campaign ID at the **validated sha** (`{REPO_SLUG}-{VALIDATED_SHA7}-{NNN}`, numbering continuing where the baseline's findings at that sha leave off), carry it on its ledger event (`event.finding`) rather than writing the baseline — gate A15 reserves that for the three audit skills — with `origin: "validate-findings"`, a `finding_identity` fingerprint, and this validation report's path in `source_findings` — the validation report is **class-1 (execution-verified) evidence** for it. The finding enters at `validation_status: not_verified` like every routed append; its execution-confirmed validity then flows through the standard ledger path (python3 -m traust.cli ledger emit-validation / `/track-findings`), where the E0/E1 evidence-grade rules decide whether it sets `confirmed`. Distinct from the P5 discovery-sweep *candidates* above (`origin: validation-discovery`): those are unexecuted hypotheses and still go to `/triage` generic ingest first — only executed, verdict-carrying novel findings route directly. - `execution_log_ref`: relative path to `validation-audit.jsonl`. - `negative_results[]`: probes that ran clean (reuse `report.schema.json#/$defs/negative_result`). The Markdown render mirrors the structure of `*-security-audit.md`: executive summary table, per-finding verdict cards with embedded evidence excerpts, an attack-chain section with ASCII path diagrams, a novel-findings section formatted identically to audit findings, and an appendix linking every artifact. --- ## Credential-liveness verification (gitleaks candidates) When the findings under validation include committed-credential candidates (a `-gitleaks.json` artifact from python3 -m traust.cli adapters gitleaks, or secret findings whose evidence cites one), run the dedicated verifier instead of hand-probing: ```bash python harnessing/5-validate/validate-findings/credential_liveness.py \ --candidates -gitleaks.json --repo \ --targets targets.yaml [--endpoint openshift=https://api.…:6443] ``` - **One read-only introspection call per credential** (GitHub `/user`, GitLab `/api/v4/user`, Slack `auth.test`, OpenShift `users/~`, AWS `sts get-caller-identity`) → `CONFIRMED_LIVE` / `REVOKED` / `UNTESTABLE(reason)`. Never a mutation: revocation and rotation are the finding owner's actions. - **Explicit-only scope**: the `credential` adapter unlocks solely via `targets.yaml#credential_probes.classes` (see `targets.example.yaml`). Inline/inferred scope, `--auto`, and `--destructive` cannot unlock it; without a targets file every candidate verdicts UNTESTABLE. `--dry-run` plans probes (with scope decisions) without touching anything. - **Secret hygiene**: the verifier re-reads credentials from the checkout (the candidates artifact is secret-free by construction), keeps them in memory only, masks to a 4-char prefix in outputs, and refuses to write an artifact that would contain a recovered secret. Its audit trail (`credential-liveness-audit.jsonl`) is append-only like `validation-audit.jsonl`. - **Into the report**: fold each probed candidate into `validated_findings[]` — `CONFIRMED_LIVE` ⇒ `verdict: confirmed` with the introspection response (status + identity) as class-1 evidence; `REVOKED` ⇒ `verdict: refuted` **scoped to exposure-now only** (the historical exposure finding stands — a secret that lived in git history was still exposed for its lifetime; say so in `deviation_from_claim`); `UNTESTABLE` ⇒ `verdict: inconclusive` with the reason. --- ## Bundled package and benchmark assets [__init__.py](__init__.py) declares this package’s public module surface. The [benchmark expectations](benchmark/expected.json) pair the safe and vulnerable manifests in [benchmark/fixtures/](benchmark/fixtures/). `tests/test_target_attestation.py` loads these expectations through `traust_engine.sweep.benchmark.load_expected` to verify scorer semantics. Consult the benchmark assets when maintaining the scorer; live execution still requires the scope and review gates below. ## Safety Controls - **Default-deny scope**: no action runs without a positive scope match. Inferred scope never includes control-plane namespaces. - **Staged by default**: human reviews the plan unless `--auto`. `--auto` is refused if `metadata.approval.environment` is not `lab` in `targets.yaml`. - **Destructive opt-in**: steps that delete data, kill pods, exhaust resources, or cannot be rolled back require `--destructive` AND must not match `off_limits`. - **Rollback**: `apply-manifest` and `create-cr` auto-delete what they created; `patch-cr` captures pre-image and restores it; `exec`/`network-probe` have no rollback (classified accordingly). - **Audit trail**: `validation-audit.jsonl` is append-only; the report embeds its sha256 so tampering is detectable. - **Expiry**: `targets.yaml#expires` in the past aborts before Phase 1. - **Listener isolation**: SSRF/callback probes bind a listener on `127.0.0.1` and reach it via `kubectl port-forward` reverse — no external egress required or permitted. --- ## Output Layout ``` /findings/// ├── -security-audit.{json,md} # existing ├── -threat-model.md # existing ├── -triage.{json,md} # existing ├── -attack-plan.yaml # new — reviewed plan ├── -validation.json # new — schema-validated ├── -validation.md # new — rendered ├── validation-audit.jsonl # new — append-only action log └── artifacts/ # new — evidence files ``` ## Integrations Consumed artifacts (producer named per artifact): - `*-security-audit.json` (`/secure-code-audit`), `*-threat-model.md` (`/threat-model`), `*-triage.json` (`/triage`) — the findings sources ingested in Phase 1. - `targets.yaml` rules-of-engagement — human-authored scope binding. - `*-gitleaks.json` — produced by python3 -m traust.cli adapters gitleaks; input to the credential-liveness verifier. - `*-refuted-register.json` — produced by `/track-findings`; scopes re-validation of refuted findings. Emitted artifacts: - `*-validation.json` — consumed by `/track-findings` via python3 -m traust.cli ledger emit-validation (execution-verified ledger events) and by `/validation-fuzz-dashboard`; class-1 evidence for routed novel findings (Phase 5 routing statement above). - `*-validation.md`, `validation-audit.jsonl`, `artifacts/` (incl. replay scripts), `target-attestation.json` — human/countersign companions; the attestation gates the ledger emitter. - Baseline appends for executed novel findings (campaign IDs at the validated sha, `origin: "validate-findings"`) — consumed by `/track-findings` / `build_cumulative.py` and the dashboards. - Discovery-sweep candidates (`origin: validation-discovery`) — consumed by `/triage` generic ingest. ## Spend declaration (calibration tuple) After this skill's report/artifact is written, declare the run's spend against the target so `estimate_scan` can calibrate per-skill cost models (contract: docs/model-routing.md; analysis: progress-tracker/metrics/estimate-calibration-analysis.md §F5): ```bash python3 -m traust.cli registry models spend --skill validate-findings \ --model [--tokens-in ] [--tokens-out ] \ --repo --loc [--batch ] ``` Token counts are OPTIONAL and best-effort: pass them when the orchestrator has them (Task results carry per-subagent usage), otherwise omit them — an agent cannot observe its own usage mid-run. **This row is a routing marker, not a cost claim**; actual per-lane cost is attributed from session transcripts by python3 -m traust.cli metrics attribute-spend. Never skip the row: an unattributed run is a calibration gap.