25 AI Agent Evidence Pack Spec

AI Agent Evidence Pack Spec Skill

Use When

• Use when defining AI-agent compliance evidence items, sources, collection, sampling, integrity, chain of custody, retention, redaction, presentation, and auditor access. Use evidence-pack-builder to assemble a specific pack and control packs to define obligations.

Do Not Use When

• Do not use as the incident evidence pack spec — this is the steady-state compliance evidence pack; the incident pack is a superset for SEV3+ events. Cross-link to 06-deployment-operations/17-ai-incident-evidence-pack-spec.
• Do not use this skill to fabricate missing project facts, legal conclusions, test results, approvals, or certification claims.

Required Inputs

Capability and permission boundaries

Default to read-only. Read and search access to the supplied artefacts are required. Editing is limited to an explicitly authorised requested draft or project files. Execute validation only when authorised; publishing, signature, certification, production mutation, destructive action, spending, and risk acceptance require explicit authority.

Degraded Mode

When files, tools, network, rendering, fonts, execution, or evidence are unavailable, return the narrowest useful qualified draft or finding set. Name every unavailable check and its consequence; an unassessed check is never a pass. Preserve evidence already gathered and provide the exact next verification step.

Decision Rules

Workflow

1. Confirm the requested artefact, audience, scope, decision owner, and applicable baseline or version. Work read-only by default; source mutation, publication, signature, certification, production change, or risk acceptance requires explicit authority.
2. Inspect every required input and record missing, stale, conflicting, or inaccessible evidence. Stop claims that depend on an unresolved required input.
3. Apply the Decision Rules, then execute the existing Core Instructions below in order; preserve project terminology and trace each material statement to its source.
4. Test the draft against the output acceptance conditions and domain quality standards. If a check cannot run, mark it not assessed and never convert it into a pass.
5. On failure, recover by preserving completed evidence, identifying the narrowest corrective action and owner, and rerunning only the affected checks before handoff.
6. Produce the named artefact and evidence record; publish, sign, certify, mutate production, or accept risk only under explicit authority.

Outputs

Evidence Produced

Quality Standards

• Every control row in every control pack shall map to one or more evidence items. Every evidence item shall name source system, collector, capture method, frequency, retention, sampling, redaction, presentation. Chain-of-custody shall be tamper-evident. Auditor portal access shall be time-bound, logged, and named-recipient.
• Use deterministic acceptance conditions and preserve traceability from source to decision and output.
• Separate facts, inferences, assumptions, and approvals; never present one as another.
• Apply 28-anti-ai-slop during authoring and 29-ai-slop-audit at major checkpoints and release.

Anti-Patterns

• Producing AI Agent Evidence Pack Spec from assumptions instead of named project sources. Fix: Cite the source or mark the item unverified.
• Treating a missing or inaccessible check as passed. Fix: Mark it not assessed, state impact, and block dependent claims.
• Using vague gates such as adequate, secure, or user-friendly. Fix: Replace each with an observable criterion, threshold, and evidence source.
• Copying a generic template without product, control, role, version, or jurisdiction detail. Fix: Ground every section in the supplied context and remove unused boilerplate.
• Publishing, signing, certifying, changing production, or accepting risk without authority. Fix: Prepare a draft and route the decision to the accountable owner.
• Listing evidence without provenance or an acceptance result. Fix: Record source, period, integrity check, mapping, and pass/fail/not-assessed status.

Worked Example

Example: if provenance, integrity, period, or control mapping is absent, quarantine the item and record the gap. Record the evidence and result in the validation record; this avoids misleading or tampered evidence.

References

• Generation logic: load when creating the complete artefact.
• Skill notes: consult for local examples and invocation context.
• Repository operating rules: apply the engine's routing, evidence, and release gates.

Overview

The evidence pack is the auditor's reading set. The SOC 2 / ISO / HIPAA control packs name what evidence is required; this spec defines how it is collected, where it lives, how it is sampled, how it is redacted, how it is presented, and how the auditor accesses it. The software-dev pass owns the collectors; this spec defines the contract.

Quick Reference

Core Instructions

Step 1: Evidence items per control class

Enumerate evidence per control class (governance, access, monitoring, change, incident, supplier, privacy, integrity, availability, confidentiality, processing integrity, BAA). Each item carries:

• Artefact name and format.
• Source system (e.g., orchestrator, dispatcher, IAM provider, observability platform).
• Collector (cross-link software-dev pass collector name).
• Capture method (automated push, scheduled job, sign-off-ledger entry).
• Frequency (continuous, daily, weekly, monthly, quarterly, annual, on-event).
• Retention (per evidence-pack retention schedule).
• Sampling (full population, 25 stratified, 5% sample, etc.).
• Redaction class.
• Presentation format (CSV, signed JSON, PDF, screenshot, log export).

Step 2: Pack layout

evidence-pack-<window>/
  manifest.json
  policies/
    agent-action-governance-policy.pdf
    ...
  controls/
    soc2/
      CC1-1.md
      CC6-1/
        access-review-Q1.csv
        access-review-Q2.csv
        ...
    iso27001/
      A-8-15.md
      ...
    hipaa/
      164-312-b.md
      ...
  evidence/
    audit-log-integrity-reports/
    kill-switch-drill-reports/
    approval-events-sample-25.csv
    daily-review-tickets-sample-25.csv
    pr-sample-25.csv
    sub-processor-list.json
    baa-ledger.csv
    dpa-ledger.csv
  incidents/
    SEV1/
    SEV2/
  signed-zip.zip
  signed-zip.zip.sha256

Step 3: Sampling protocol

Sampling rationale follows AICPA AT-C 205 and ISO/IEC 17021 sampling guidance. Stratification keys: feature, tier, severity, tenant size, time-of-window quintile.

Step 4: Chain-of-custody

• Each artefact captured to immutable storage with content hash (SHA-256 default).
• Capture event names: source system, time window, capture operator or job id, hash, storage URI.
• Modifying a captured artefact is impossible by design; corrections create a derived artefact with back-link.
• All captures listed in manifest.json with their hashes; the manifest itself is signed at pack close.

Step 5: Retention

Per the Audit-Log Retention Policy and per the SOC 2 / ISO / HIPAA retention requirements:

HIPAA minimum: 6 years from creation or last effective date. SOC 2: typical 5 years; bias toward 7. ISO: as policy.

Step 6: Redaction policy

Same redaction classes as the incident evidence pack spec:

• Other tenants' data: redact by default; not overridable.
• User PII fields: redact unless regulator requires unredacted under order.
• Provider-confidential under NDA: redact unless provider authorises.
• Internal pricing: redact for customer view; exec sign-off for unredacted to auditor.

Redaction events appended to the manifest.

Step 7: Presentation format

• Markdown for narrative artefacts (policies, control narratives, SoA, walkthrough scripts).
• CSV or signed JSON for tabular evidence (access reviews, approval events, tickets, drill reports).
• PDF for signed documents (policies, sign-off ledger, BAAs).
• Screenshots / video for in-product disclosures, drills, demos.
• Signed zip for the bundle as a whole.

Step 8: Auditor portal access governance

• Access is time-bound (default: audit window + 14 days).
• Named-recipient only; no shared accounts.
• Every access is logged with name, IP, time, artefact viewed.
• Download events recorded with hash of downloaded artefact.
• Portal access revoked on day +1 after final report.

Step 9: Write the spec

AI_Agent_Evidence_Pack_Spec.md sections: 1) Evidence items per control class, 2) Pack layout, 3) Sampling protocol, 4) Chain-of-custody, 5) Retention, 6) Redaction policy, 7) Presentation format, 8) Auditor portal access governance, 9) Cross-refs, 10) Sign-off.

Standards

• AICPA TSP 100; AICPA AT-C 205
• ISO/IEC 27007 (auditing guidelines)
• ISO/IEC 17021 (certification body process)
• HIPAA §164.316(b) (documentation retention)
• HHS OCR audit protocol

Resources

• logic.prompt, README.md, references/ai-agent-attestation-evidence-pack-template.md, references/ai-agent-evidence-frequency-table.md.