Validate Findings

Validate Findings — Live Validation & Attack-Chain Harness

Paths. analysis-results/… and progress-tracker/… in this skill are the default workspace layout. They resolve through locations.yaml in $TRAUST_CONFIG_HOME (docs/setup.md, Storage locations); substitute your configured roots.

Take the static outputs of secure-code-audit, threat-model, and triage and prove or refute them against a live authorized environment. Then go further: chain confirmed findings into multi-step kill-chains and hunt for novel attacks the static analysis missed.

Authorization: This skill executes potentially state-changing actions against live infrastructure. It MUST only be run against targets you own or have explicit written authorization to test. Every action is gated by the scope guard (scope.py) — out-of-scope steps are refused and logged, never silently skipped.

Input

$ARGUMENTS is parsed as whitespace-separated tokens. The first non-flag token is the findings source; remaining tokens are flags.

Findings source (required — one of)

Refuted findings stay in scope. When a <repo>-refuted-register.json sits next to the audit report (emitted by python3 -m traust.cli ledger emit-triage), its entries are candidate targets, not exclusions: a finding dismissed as a false positive — even human-countersigned — is a falsifiable claim, and a reproducing exploit from this skill overrides the assertion via evidence-class precedence (fp_overridden, loudly surfaced with the countersigner attributed). Include register entries in attack planning when scope and budget allow.

Impact-analysis artifacts as target selection. When analysis-results/impact/<cve>-impact-analysis.json (from /impact-analysis) covers the finding's CVE, use it two ways: (a) prioritization — repos classified affected with triage's attacker_influence: plausible are the validation candidates worth cluster time first; (b) attack planning — the repo's evidence.govulncheck_trace names the call path caller-first, so the first in-repo frame identifies which interface the attack plan should drive. A confirmed exploitation here is execution evidence (class 1): it flips the finding to confirmed in the disposition ledger and is the terminal answer to "is this repo actually affected".

Product packages (*-findings/ directories) contain sub-directories grouped by repo or by sub-group/repo (1–2 levels). The harness auto-detects the package layout: if the source directory has no audit reports at the top level but has sub-directories with audit reports, it is treated as a package and all repos are ingested. Findings from each repo carry a source_repo tag for traceability in the plan, execution log, and validation report.

Scope binding (at least one mode; modes are additive)

If no scope mode is given, the harness runs --dry-run implicitly and warns.

Execution control

Scope Binding — Resolution & Enforcement

Run python harnessing/5-validate/validate-findings/scope.py semantics:

1. Load mode-1 file if --targets given; merge mode-2 inline flags on top; if --infer-scope, call ingest.infer_scope() and merge with lowest precedence.
2. Compile into a Scope object exposing is_in_scope(action: Action) -> (bool, reason) where Action = {adapter, verb, context, namespace, resource, name, image, extra}.
3. Hard denies (always refused regardless of flags or --auto --destructive):
   • Any entry in targets.yaml#off_limits.
   • Any namespace matching kube-system, openshift-etcd, openshift-kube-apiserver*, openshift-authentication* unless explicitly listed in mode-1 clusters[].namespaces.
   • expires date in the past.
4. Preflight every bound target via the adapter's preflight() to capture a fingerprint (cluster version, node count, image digest, WASM sha256). The fingerprint goes into validation.json#metadata.target_fingerprint so results are reproducible.

Every executed step is logged with the scope-check outcome. A blocked_by_scope verdict is a result, not an error — it tells the reader the PoC would have crossed a boundary the engagement does not permit.

Benchmark mode (P7)

analysis-results/scan-testing/validation-benchmark/benchmark-findings.json (from python3 -m traust.cli sweep benchmark plan) is a first-class input: validate its claims against the deployed fixtures exactly like campaign findings, then score the report with run_validation_benchmark.py score --variant vuln|safe. Benchmark runs live in the harness-QA tree and never touch campaign metrics.

Phase 0b — Target attestation (fail-closed, P2)

After scope loads and before ANY probe executes, attest the target and write target-attestation.json into the validation output dir:

python3 -m traust.cli admin attest-target --out <out-dir>/target-attestation.json \
    --namespace <operand-ns> [--csv <operator-csv>] [--selector <pods>] \
    [--version <deployed> --affected-range '<vX.Y.Z'] \
    [--kubeconfig <path>] [--context <ctx>]

Record the summary in metadata.target_attestation. If attested: false: probes MAY still run for diagnosis, but every verdict in the report is structurally void — emit_validation_ledger_events routes the entire run to needs_review as environment_invalid (or attestation_missing for post-0.176.0 reports without the artifact). The rhoso.v2 failure class — 34 false "refuted" from a never-installed operator — becomes 0 verdicts at the front door.

Phase 0c — Positive-control pairing (assay validity, P1)

Every refutation-capable probe step carries paired positive controls in its plan entry, executed in the same session and recorded in the step's controls[] (schema positive_control):

Rules (enforced by the soundness gate + ledger emitter):

• A refuted verdict with a failed control is quarantined as failed-positive-control — any report age: the assay demonstrably didn't work.
• A refuted verdict with no passing control, on reports at/after harness 0.177.0, quarantines as missing-positive-control.
• Controls never substitute for the probe: they prove the assay could have detected the claim, converting "absence of evidence" into "evidence of absence, with the assay proven live."

Phase 0d — Differential probing for authz claims (P3)

For every "role/subject can(not) do X" refutation, probe the pair in the same session and record it in the step's differential block:

• the claimed action, AND
• a neighbor action with a known-different expected outcome (same subject → a verb it IS allowed; same verb → a subject that IS authorized).

If both outcomes are identical, the oracle cannot discriminate allowed from denied — the refutation is unsound regardless of which way it pointed (non-discriminating-oracle, quarantined at any report age). Post-0.178.0, an authz-class refutation without a discriminating differential quarantines as missing-differential-probe. One extra request per probe; catches wrong-oracle failures without knowing why the oracle is wrong.

Phase 0e — Severity validation on confirmations (P9)

Every confirmed finding records severity_validation: the CVSS components the exploit ACTUALLY demonstrated (attack vector used, privileges the probe identity held at success, user interaction, scope crossing observed from the E0 artifact, impact axes evidenced) and the signed delta vs the claimed score. When |delta| >= 1.0 and the evidence grade is E0/E1/E2 (never E3), emit a proposal — the emitter routes it to the countersign severity decision as a needs_review item; machines never write disposition.severity. A demonstrated DOWNGRADE ("exploitation required cluster-admin") is exactly as valuable as an upgrade. Findings that are constituents of a demonstrated attack chain also record chain_context (chain id, chain severity, role) — chain membership is severity evidence and belongs in the proposal rationale.

Phase 6 pointer — discovery sweeps & replay artifacts (P5/P8)

Replay (P8, required on every probe): record a self-contained replay script per executed step in artifacts/replay/<step_id>.sh (+ inputs file when the probe posts data) and reference it in the step's replay block with the attestation fingerprint sha. A countersign human re-runs the exact probe instead of trusting the transcript.

Discovery sweeps (P5, optional phase after replay/chained/novel): outputs are NEW finding candidates with origin: validation-discovery, routed to /triage generic ingest — never straight to the ledger:

1. State diffing (implemented): python3 -m traust.cli impact cluster-state-diff snapshot before the sweep and after it; diff turns unexpected deltas (new/mutated RBAC, SCC changes, webhook mutations, deleted NetworkPolicies, new exposure) into candidates, excluding declared probe side-effects (--expected).
2. Anonymous-surface sweep — unauthenticated reachability vs an authenticated baseline (per-sweep increment, not yet implemented).
3. Privilege-escalation chain search — seeded from operator-priv-profile + the RBAC graph; attempt the cheapest link under the scope guard (not yet implemented).
4. Browser role×route matrix + IDOR probing (browser lane; not yet implemented).

All sweeps inherit the hard scope guard unchanged.

Phase 1 — Ingest & Normalize

python harnessing/5-validate/validate-findings/ingest.py <findings-source> > /tmp/normalized.json

1. Locate the three report types in the source directory. JSON is preferred; fall back to Markdown parsing when JSON is absent.
2. Build a unified Finding list. For each finding merge:
   • security-audit: id, title, severity, cwes, locations, evidence[], attack_pattern, cvss
   • triage (if present): verdict, verify_verdict, confidence, preconditions[], first_links[], severity_label, owner_hint
   • threat-model (if present): linked threat_id, entry_point, asset, controls
3. Build a ThreatGraph skeleton: assets (with sensitivity), entry points (with trust level + reachable assets), threats (with status + linked finding evidence).
4. Extract embedded PoCs: scan evidence[] code blocks and attack_pattern text for fenced yaml/bash/json/curl blocks and the triage rationale STEP narratives. Tag each as {lang, body, source_field}.

Findings with triage verdict == false_positive are carried through but default to technique: skip in the plan (override with --include-fp). Findings with triage verdict == needs_review (evidence not statically locatable; never proven false) are not skipped — they are planned like true_positive and the resulting step is tagged triage_was_needs_review: true.

Phase 2 — Attack Planning

python harnessing/5-validate/validate-findings/plan.py /tmp/normalized.json --scope <scope.json> > <out>/<repo>-attack-plan.yaml

The plan is an ordered list of steps, each:

- id: step-007
  finding_ref: ODR-2026-004        # or novel_ref / chain_ref
  technique: replay                # replay | adapted | chained | novel | recon
  adapter: k8s
  verb: apply-manifest
  target: {context: lab-spoke-1, namespace: ramen-ops}
  payload: |                       # the literal manifest / command / wasm-invoke
    apiVersion: ramen.openshift.io/v1alpha1
    kind: Recipe
    ...
  classification: mutating         # safe | mutating | destructive (from adapter.classify)
  expected: "pods/exec succeeds in namespace outside tenant scope"
  rollback: "kubectl --context lab-spoke-1 -n ramen-ops delete recipe pwn"
  preconditions: [step-003]        # steps that must succeed first

2a. Replay steps

For each non-FP finding, convert extracted PoCs into one or more steps:

2b. Chained steps (chain.py)

Build the attack graph and emit one chain_ref step-group per discovered path:

• Nodes: findings, threat-model entry points, threat-model assets, recon-discovered resources (added in Phase 4 second pass).
• Edges:
  • entry_point → finding when the finding's preconditions are satisfiable at that entry point's trust level.
  • finding_A → finding_B when A's expected impact (token theft, ns escape, RBAC grant, file write, network position) satisfies any of B's preconditions. Use the capability vocabulary in chain.py:CAPABILITIES.
  • finding → asset when the finding's impact references the asset.
• Search: DFS from every entry point to every asset with sensitivity ≥ high. Deduplicate by node-set. Cap path length at 6.
• For each chain, emit ordered steps reusing the replay steps where they exist, inserting technique: chained glue steps (e.g. "extract SA token from step-003 evidence and set KUBECONFIG for step-004").
• Annotate with MITRE ATT&CK technique IDs from the shared pinned tables (harnessing/attack-coverage/tables/, loaded via attack_refs.capability_map(); IDs validated against the pinned ATT&CK release at import).

2c. Novel steps (novel.py)

1. Recon plan: emit technique: recon steps for the bound target type(s) — these are always classification: safe:

   • k8s/operator/component: kubectl get crd,clusterrolebinding,rolebinding,networkpolicy,validatingwebhookconfiguration,mutatingwebhookconfiguration,svc,ep -A -o json; kubectl auth can-i --list --as=<each in-scope SA>; kubectl get pods -o jsonpath for securityContext, hostPath, SA mounts, capabilities.
   • container: inspect; cat /proc/self/status (CapEff), /proc/mounts, env, listening sockets.
   • wasm: wasm-tools dump --imports --exports; component-model world inspection.

2. Diff recon output against threat-model entry_points + controls. Surfaces present live but absent from the model are candidate novel entry points.

3. Crypto posture probes — when findings include category: cryptography or pqc_classification and a cluster context is bound, run runtime data collection:

   python3 -m traust.cli adapters crypto-audit cluster --context <ctx> --namespaces <ns> [--discover] [--groups "..."]
   

   This emits crypto-audit/v1 (tier: runtime) with raw TLS negotiation data, crypto-policy state, and OpenSSL/Go versions. For source and image tier crypto analysis, delegate to the crypto-analysis skill.

   These facts feed the PQC PROBE_CATALOGUE entries and chain.pqc_caps_from_probe() for capability mapping. Record each run in metadata.tools and deterministic_steps.

4. Pattern probes — for each candidate, emit a probe step from the catalogue in novel.py:PROBE_CATALOGUE keyed by target type and surface class:

   • Operator: confused-deputy (cross-ns *Ref fields in CRD schema), SSRF (URL-typed CRD fields → harness listener), TOCTOU (status-subresource race), webhook bypass (label/annotation that skips admission), over-broad RBAC (auth can-i --list shows * * *).
   • Pod / container: writable hostPath, CAP_SYS_ADMIN/CAP_NET_ADMIN presence, /var/run/docker.sock or CRI socket mount, SA token with cluster-admin reachability, procfs breakout, shared PID/net namespace.
   • Cluster component: unauth metrics/pprof/healthz with side-effects, anonymous-auth on kubelet, insecure-port residue, etcd client cert reuse.
   • WASM: unrestricted WASI preopen dirs, hostcall import without capability check, linear-memory OOB on exported function, missing fuel/epoch limits.

5. AI-assisted hypotheses — after recon executes (Phase 4 first pass), reason over: recon output, unexploited threat-model threats (status: unmitigated with no linked finding), and the CWE distribution of confirmed findings. Propose up to --max-novel additional attack hypotheses. Each becomes a technique: novel step with a clear expected outcome that distinguishes confirmed from refuted.

2d. Ordering & classification

Order: all recon → all replay (severity desc) → chained → novel. Run adapter.classify() on every step; if destructive and --destructive not set, mark skip: destructive-not-permitted.

Write the plan to <out>/<repo>-attack-plan.yaml.

Phase 3 — Review Gate

Unless --auto or --dry-run:

1. Print a summary table: step count by {technique × classification}, every mutating/destructive step's verb + target, and every step that scope.is_in_scope() would refuse.
2. Ask the user to approve. Accept: yes (run all permitted), edit (open attack-plan.yaml, re-read, re-summarize), skip <id...> (mark listed steps not_attempted: user-skip), or no (exit, plan kept on disk).
3. Record the approval (who, when, mode) in validation.json#metadata.approval.

If --dry-run: stop here. The plan on disk is the deliverable.

Phase 4 — Execute

python harnessing/5-validate/validate-findings/execute.py <out>/<repo>-attack-plan.yaml --scope <scope.json> --out <out>

For each step in plan order:

1. scope.is_in_scope(step.action) — if false, record verdict: blocked_by_scope with reason; continue.
2. Check preconditions — if any referenced step did not yield verdict: confirmed, record verdict: not_attempted with reason precondition-failed:<id>; continue.
3. Dispatch to TARGET_ADAPTERS[step.adapter].execute(step, scope, audit):
   • Capture stdout/stderr/exit, plus adapter-specific evidence (resource YAML before/after, HTTP response body, WASM trap).
   • Write artifacts to <out>/artifacts/step-<id>.* and compute sha256.
   • Compare observed outcome to step.expected → verdict ∈ {confirmed, refuted, inconclusive}.
   • Refutation-soundness gate (soundness.py, applied deterministically by execute.py — never bypass it when composing verdicts by hand): refuted is un-emittable when (a) the probe transcript matches an error signature (Forbidden/validator-RBAC denial, command not found, jsonpath errors, NotFound, does not exist, Could not connect, an unsubstituted {ns}/{sa} template), (b) an RBAC-style probe enumerated zero concrete subjects (empty vf-rbac-tried: list) or carries a literal system:serviceaccount:{ns}:{sa} placeholder, or (c) the run directory contains install-failure.yaml (whole run — the operand never deployed). Gated verdicts downgrade to inconclusive and carry a machine-readable soundness_flag (error-signature:<name> | rbac-zero-subjects | rbac-template-placeholder | target-not-deployed). Downstream, flagged findings route to needs_review — never to the false-positive/countersign path (emit_validation_ledger_events.py).
4. Append one JSONL line to <out>/validation-audit.jsonl:

   {"ts":"<iso8601>","step_id":"...","adapter":"...","verb":"...","target":"...","scope_check":"pass","classification":"...","verdict":"...","evidence":["artifacts/step-007.log"],"rollback":"..."}
   

5. If classification: mutating and the adapter provides rollback(), run it immediately after evidence capture and record rollback_performed: true|false with output.
6. Two-pass for novel: after recon steps complete, re-invoke novel.py with live recon artifacts to materialize the AI-assisted hypotheses (Phase 2c-4) into concrete steps, append them to the plan, and continue execution.

Rate-limit: ≤ 1 mutating step per 2 s per cluster context; back off ×2 on 429/503 from the API server.

Phase 5 — Report

python harnessing/5-validate/validate-findings/report.py <out> > <out>/<repo>-validation.json
python3 -m traust.cli reporting validate --schema contracts/schemas/validation.schema.json <out>/<repo>-validation.json
python harnessing/5-validate/validate-findings/report.py <out> --markdown > <out>/<repo>-validation.md

<repo>-validation.json conforms to contracts/schemas/validation.schema.json:

• metadata: date, harness_version (VERSION + short SHA), scope_binding_mode (explicit|inline|inferred|mixed), target_fingerprint[], approval, flags.
• source_reports[]: paths + sha256 of the ingested audit/threat-model/triage files.
• summary: counts by verdict, by technique, novel-finding count, highest-impact confirmed chain.
• validated_findings[]: one per source finding — source_id, verdict, technique, evidence[], observed_impact, deviation_from_claim, rollback_performed, plus soundness_flag when the refutation-soundness gate downgraded a refuted to inconclusive (see Phase 4; report.py re-applies the gate as a backstop at rollup).
• attack_chains[]: chain_id, name, entry_point, terminal_asset, mitre_attack_refs[], steps[], verdict.
• novel_findings[]: full report.schema.json#/$defs/finding objects + discovery_method + chain_context — ready to feed into file-security-defect.

Routing novel findings (user directive 2026-07-27 — direct ledger entry, no /triage precondition). A novel finding discovered by live validation is a first-class campaign finding at birth, same convention as /vuln-scan supplements and /verify-remediation regression routing (python3 -m traust.cli route regressions is the reference mechanics): mint its campaign ID at the validated sha ({REPO_SLUG}-{VALIDATED_SHA7}-{NNN}, numbering continuing where the baseline's findings at that sha leave off), carry it on its ledger event (event.finding) rather than writing the baseline — gate A15 reserves that for the three audit skills — with origin: "validate-findings", a finding_identity fingerprint, and this validation report's path in source_findings — the validation report is class-1 (execution-verified) evidence for it. The finding enters at validation_status: not_verified like every routed append; its execution-confirmed validity then flows through the standard ledger path (python3 -m traust.cli ledger emit-validation / /track-findings), where the E0/E1 evidence-grade rules decide whether it sets confirmed. Distinct from the P5 discovery-sweep candidates above (origin: validation-discovery): those are unexecuted hypotheses and still go to /triage generic ingest first — only executed, verdict-carrying novel findings route directly.

• execution_log_ref: relative path to validation-audit.jsonl.
• negative_results[]: probes that ran clean (reuse report.schema.json#/$defs/negative_result).

The Markdown render mirrors the structure of *-security-audit.md: executive summary table, per-finding verdict cards with embedded evidence excerpts, an attack-chain section with ASCII path diagrams, a novel-findings section formatted identically to audit findings, and an appendix linking every artifact.

Credential-liveness verification (gitleaks candidates)

When the findings under validation include committed-credential candidates (a <repo>-gitleaks.json artifact from python3 -m traust.cli adapters gitleaks, or secret findings whose evidence cites one), run the dedicated verifier instead of hand-probing:

python harnessing/5-validate/validate-findings/credential_liveness.py \
    --candidates <repo>-gitleaks.json --repo <checkout> \
    --targets targets.yaml [--endpoint openshift=https://api.…:6443]

• One read-only introspection call per credential (GitHub /user, GitLab /api/v4/user, Slack auth.test, OpenShift users/~, AWS sts get-caller-identity) → CONFIRMED_LIVE / REVOKED / UNTESTABLE(reason). Never a mutation: revocation and rotation are the finding owner's actions.
• Explicit-only scope: the credential adapter unlocks solely via targets.yaml#credential_probes.classes (see targets.example.yaml). Inline/inferred scope, --auto, and --destructive cannot unlock it; without a targets file every candidate verdicts UNTESTABLE. --dry-run plans probes (with scope decisions) without touching anything.
• Secret hygiene: the verifier re-reads credentials from the checkout (the candidates artifact is secret-free by construction), keeps them in memory only, masks to a 4-char prefix in outputs, and refuses to write an artifact that would contain a recovered secret. Its audit trail (credential-liveness-audit.jsonl) is append-only like validation-audit.jsonl.
• Into the report: fold each probed candidate into validated_findings[] — CONFIRMED_LIVE ⇒ verdict: confirmed with the introspection response (status + identity) as class-1 evidence; REVOKED ⇒ verdict: refuted scoped to exposure-now only (the historical exposure finding stands — a secret that lived in git history was still exposed for its lifetime; say so in deviation_from_claim); UNTESTABLE ⇒ verdict: inconclusive with the reason.

Bundled package and benchmark assets

init.py declares this package’s public module surface. The benchmark expectations pair the safe and vulnerable manifests in benchmark/fixtures/. tests/test_target_attestation.py loads these expectations through traust_engine.sweep.benchmark.load_expected to verify scorer semantics. Consult the benchmark assets when maintaining the scorer; live execution still requires the scope and review gates below.

Safety Controls

• Default-deny scope: no action runs without a positive scope match. Inferred scope never includes control-plane namespaces.
• Staged by default: human reviews the plan unless --auto. --auto is refused if metadata.approval.environment is not lab in targets.yaml.
• Destructive opt-in: steps that delete data, kill pods, exhaust resources, or cannot be rolled back require --destructive AND must not match off_limits.
• Rollback: apply-manifest and create-cr auto-delete what they created; patch-cr captures pre-image and restores it; exec/network-probe have no rollback (classified accordingly).
• Audit trail: validation-audit.jsonl is append-only; the report embeds its sha256 so tampering is detectable.
• Expiry: targets.yaml#expires in the past aborts before Phase 1.
• Listener isolation: SSRF/callback probes bind a listener on 127.0.0.1 and reach it via kubectl port-forward reverse — no external egress required or permitted.

Output Layout

<analysis-results>/findings/<product>/<repo>/
├── <repo>-security-audit.{json,md}     # existing
├── <repo>-threat-model.md              # existing
├── <repo>-triage.{json,md}             # existing
├── <repo>-attack-plan.yaml             # new — reviewed plan
├── <repo>-validation.json              # new — schema-validated
├── <repo>-validation.md                # new — rendered
├── validation-audit.jsonl              # new — append-only action log
└── artifacts/                          # new — evidence files

Integrations

Consumed artifacts (producer named per artifact):

• *-security-audit.json (/secure-code-audit), *-threat-model.md (/threat-model), *-triage.json (/triage) — the findings sources ingested in Phase 1.
• targets.yaml rules-of-engagement — human-authored scope binding.
• *-gitleaks.json — produced by python3 -m traust.cli adapters gitleaks; input to the credential-liveness verifier.
• *-refuted-register.json — produced by /track-findings; scopes re-validation of refuted findings.

Emitted artifacts:

• *-validation.json — consumed by /track-findings via python3 -m traust.cli ledger emit-validation (execution-verified ledger events) and by /validation-fuzz-dashboard; class-1 evidence for routed novel findings (Phase 5 routing statement above).
• *-validation.md, validation-audit.jsonl, artifacts/ (incl. replay scripts), target-attestation.json — human/countersign companions; the attestation gates the ledger emitter.
• Baseline appends for executed novel findings (campaign IDs at the validated sha, origin: "validate-findings") — consumed by /track-findings / build_cumulative.py and the dashboards.
• Discovery-sweep candidates (origin: validation-discovery) — consumed by /triage generic ingest.

Spend declaration (calibration tuple)

After this skill's report/artifact is written, declare the run's spend against the target so estimate_scan can calibrate per-skill cost models (contract: docs/model-routing.md; analysis: progress-tracker/metrics/estimate-calibration-analysis.md §F5):

python3 -m traust.cli registry models spend --skill validate-findings \
    --model <resolved model id> [--tokens-in <N>] [--tokens-out <N>] \
    --repo <target-slug> --loc <target size, if known> [--batch <batch-id>]

Token counts are OPTIONAL and best-effort: pass them when the orchestrator has them (Task results carry per-subagent usage), otherwise omit them — an agent cannot observe its own usage mid-run. This row is a routing marker, not a cost claim; actual per-lane cost is attributed from session transcripts by python3 -m traust.cli metrics attribute-spend. Never skip the row: an unattributed run is a calibration gap.