patkusch Evidence pack
Governance that cannot be evidenced does not exist as far as an auditor is concerned. The purpose of this skill is less to produce a document than to find the gap between what the organisation believes it does and what it can demonstrate — while there is still time to close it.
Run it before the audit, not during. A pack assembled under deadline documents the gaps rather than fixing them.
What counts as evidence
Ranked by how well it holds up:
- Artefacts generated by the process itself — logs, test outputs, approval records, automated reports, version history. Strongest, because they are hard to produce retrospectively.
- Contemporaneous documents — assessments, decision records, meeting minutes with dates.
- Current-state documents — policies, procedures, architecture diagrams. Necessary, but they show intent rather than operation.
- Attestation — someone says it happens. Weakest, and appropriate only where the thing genuinely leaves no trace.
An auditor's core question is not "do you have a policy?" but "show me it operating." Rank accordingly, and flag anything resting on tier 4 alone.
Assembly
1. Establish the ask
Different audiences need different packs. A certification body wants clause and control conformity; a regulator wants statutory obligations; a customer wants risk answers about their data. Ask what standard or question set applies, and scope to it. A pack that answers everything answers nothing well.
2. Validate the record before you build anything on it
python scripts/validate_record.py records/the-system.record.yaml
Do this first. An evidence pack assembled from a record that does not conform is a pack built on sand, and the validator also reports governance warnings you will otherwise have to find by hand — observed tier exceeding declared, blast radius below the tools held, untested halts, memory without provenance, expired exceptions. That last one belongs in the pack whether you like it or not, and an auditor will find it if you don't.
3. Pull from the system record
Read the record (see
framework/system-record.schema.json) and
assemble: identification and ownership, purpose and out-of-scope uses, agentic
configuration and tier history, supply-chain role, assessments with findings, incidents
with remediation status, and open exceptions with expiry dates.
4. Substantiate each claim
For every control claimed as met, locate the artefact. This is the work, and it is where the value is. Three specific things to check hard, because they are asserted far more often than they are true:
- Limits enforced outside the model. If the evidence is a system prompt, the control is not enforced. Record it as unevidenced.
- Halt mechanisms. Ask for the test record and the date. "It exists" is not evidence it works.
- Human oversight. Ask for approval records, override rates, and rejection counts. A gate with no recorded rejections invites the question of whether it is a gate — but note that a gate with a few rejections is not automatically better. Check whether the rejected items came straight back and were approved. Dissent that costs nothing is not review, and an approval rate massaged off 100% is a metric being managed rather than a control operating.
The test to apply to every claimed control
What would the system have to do to satisfy this while still causing the harm?
If the answer is "write a different sentence", the control is decorative and its
satisfaction is not evidence. evidence/adversary/ found four controls that
failed this test using nothing but ordinary engineering habit. Two of them cannot be
salvaged by better inspection and should be recorded as unevidenced whenever you meet
them:
- Scope the system declared for itself. Compliance against a self-set boundary is circular. Ask who assigned it; if the agent did, the boundary is documentation of intent.
- Provenance the system wrote about its own belief. The agent authored both the record and its justification. Provenance counts only when it points at something a third party recorded.
Auditors ask this question instinctively. Getting to it first is the entire value of assembling the pack early.
5. Report what is missing
The gap register is the most valuable page in the pack. For each: what is claimed, what evidence is absent, what would satisfy it, who owns it, and how long it takes to produce.
Some gaps can be closed before the audit. Some cannot, and for those the right move is a stated, owned remediation plan — auditors respond far better to a known gap with a plan than to a claim that collapses under a follow-up question.
6. Check the exceptions
Every exception should have an approver and an expiry. Expired exceptions are findings. An exception with no expiry is a policy change nobody wrote down, and it should be surfaced as such.
Output
Produce a directory, not a single file — auditors navigate, they do not read linearly.
evidence-pack-[system]-[date]/
├── 00-summary.md Scope, systems covered, standard addressed, gap count
├── 01-system-record.yaml The record itself
├── 02-assessments/ One file per framework assessment
├── 03-controls.md Control-by-control, each with its evidence pointer
├── 04-incidents.md History with remediation status
├── 05-gaps.md Claimed-but-unevidenced, with owners and dates
├── 06-exceptions.md Live exceptions, approvers, expiry
└── artefacts/ The evidence itself, referenced by 03
00-summary.md should state the gap count on the first page. Burying it reads as
concealment, and an auditor who finds a buried gap stops trusting the rest of the pack.
Where honesty is the strategy
The temptation is to present the strongest possible case. Resist it — not only for the obvious reason, but because an auditor who finds one overstated claim re-examines every other claim, and a pack that flags its own weaknesses buys credibility for everything else in it.
Where a control is partially implemented, say partially and say which part. Where a finding is open, give the owner and the date. Where something was assessed and found wanting, include the assessment. That posture is also the one that survives the follow-up questions, which is where packs usually come apart.