Agent governance patterns
Add governance controls to AI agent systems so tool use, content handling, delegation, rate, and accountability stay inside explicit safety and trust boundaries.
When to invoke
"Add governance to an agent that calls external tools."
"Implement policy-based access control for agent tools."
"Detect dangerous prompts before tool execution."
"Create trust scoring for a multi-agent workflow."
"Add audit trails, rate limits, content filters, or tool restrictions."
Governance flow
User Request → Intent Classification → Policy Check → Tool Execution → Audit Log
↓ ↓ ↓
Threat Detection Allow/Deny Trust Update
Apply controls before tool execution, update trust after outcomes, and write audit records for both allowed and denied actions.
Governance levels
Level
Controls
Use case
Open
Audit only, no restrictions.
Internal dev/testing.
Standard
Tool allowlist plus content filters.
General production agents.
Strict
All controls plus human approval for sensitive operations.
Financial, healthcare, legal.
Locked
Allowlist only, no dynamic tools, full audit.
Compliance-critical systems.
Control patterns
Pattern
Rule
Failure mode prevented
Policy as configuration
Store policies in YAML/JSON, not hardcoded.
Code deploy required for every policy change.
Most-restrictive-wins
Deny always overrides allow when composing policies.
A permissive policy bypasses a stricter one.
Pre-flight intent check
Classify intent before tool execution.
Dangerous action is detected only after side effects.
Tool allowlist
Permit only named tools for each role, context, or trust level.
Dynamic tool access expands beyond review.
Content filter
Block secrets, prompt injection, unsafe content, or regulated data flows.
Agent processes content it must not handle.
Human approval
Require approval for high-impact or irreversible operations.
Agent autonomously performs sensitive changes.
Rate limit
Bound requests, tool calls, tokens, or cost by actor and time window.
Runaway agent loops or cost spikes.
Trust decay
Decay trust scores over time.
Old good behavior grants permanent privilege.
Append-only audit
Never modify or delete audit entries.
Compliance evidence can be rewritten.
Fail closed
Deny action when governance checks error.
Safety system outage becomes allow-all.
Separate policy from logic
Keep enforcement independent from agent business logic.
Business code accidentally bypasses governance.
Implementation checklist
## Agent Governance Implementation Checklist
### Setup
- [ ] Define governance policy (allowed tools, blocked patterns, rate limits)
- [ ] Choose governance level (open/standard/strict/locked)
- [ ] Set up audit trail storage
### Implementation
- [ ] Add @govern decorator to all tool functions
- [ ] Add intent classification to user input processing
- [ ] Implement trust scoring for multi-agent interactions
- [ ] Wire up audit trail export
### Validation
- [ ] Test that blocked tools are properly denied
- [ ] Test that content filters catch sensitive patterns
- [ ] Test rate limiting behavior
- [ ] Verify audit trail captures all events
- [ ] Test policy composition (most-restrictive-wins)
Framework integration cues
Framework
Integration seam
PydanticAI
Wrap tools or dependencies with a policy check before invocation.
CrewAI
Gate crew tools and delegation tasks by role and trust level.
OpenAI Agents
Enforce tool allowlists and approval steps before tool calls execute.
LangChain
Wrap tools, callbacks, and chains with policy and audit middleware.
AutoGen
Gate inter-agent messages, delegation, and tool execution.
Read references/pattern-implementations.md before writing code for policy objects, classifiers, decorators, trust scoring, audit trails, or framework-specific examples.
Criteria
Gotchas
Do not hardcode policy in tool bodies : policy must be reviewable and changeable outside business logic.Do not rely on audit-only for production sensitive operations : audit after harm is not prevention.Do not let dynamic tool discovery bypass allowlists : discovered tools still need explicit policy.Do not delete audit entries : corrections should append compensating records.
Progressive disclosure and bundled resources
references/pattern-implementations.md: concrete implementations for policy, classifier, decorator, trust scoring, audit trail, and framework integration.
Environment shorthand
Use open governance for dev/testing; increase controls before production.
Output template
## Agent governance plan
**Status:** ready | implemented | blocked
**Governance level:** open | standard | strict | locked
**Agent/framework:** <name>
| Control | Decision | Evidence or implementation |
| --- | --- | --- |
| Tool allowlist | <rule> | <policy path or code seam> |
| Intent classifier | <rule> | <classifier/prompt/model> |
| Human approval | <rule> | <approval path> |
| Audit trail | <rule> | <storage/export> |
| Trust scoring | <rule> | <update/decay formula> |
### Validation
- Blocked tool denied: pass | fail
- Content filter tested: pass | fail
- Rate limit tested: pass | fail
- Audit event captured: pass | fail
Quality gate
References
1 --- 2 name: agent-governance-2 3 description: Design governance, safety, policy enforcement, trust scoring, and audit controls for AI agent systems. Use when building agents with external tools, policy-based tool access, semantic intent classification, dangerous prompt detection, multi-agent trust workflows, rate limits, content filters, or audit trails across PydanticAI, CrewAI, OpenAI Agents, LangChain, or AutoGen. 4 --- 5 6 # Agent governance patterns 7 8 Add governance controls to AI agent systems so tool use, content handling, delegation, rate, and accountability stay inside explicit safety and trust boundaries. 9 10 ## When to invoke 11 12 - "Add governance to an agent that calls external tools." 13 - "Implement policy-based access control for agent tools." 14 - "Detect dangerous prompts before tool execution." 15 - "Create trust scoring for a multi-agent workflow." 16 - "Add audit trails, rate limits, content filters, or tool restrictions." 17 18 ## Governance flow 19 20 ```text 21 User Request → Intent Classification → Policy Check → Tool Execution → Audit Log 22 ↓ ↓ ↓ 23 Threat Detection Allow/Deny Trust Update 24 ``` 25 26 Apply controls before tool execution, update trust after outcomes, and write audit records for both allowed and denied actions. 27 28 ## Governance levels 29 30 | Level | Controls | Use case | 31 | --- | --- | --- | 32 | Open | Audit only, no restrictions. | Internal dev/testing. | 33 | Standard | Tool allowlist plus content filters. | General production agents. | 34 | Strict | All controls plus human approval for sensitive operations. | Financial, healthcare, legal. | 35 | Locked | Allowlist only, no dynamic tools, full audit. | Compliance-critical systems. | 36 37 ## Control patterns 38 39 | Pattern | Rule | Failure mode prevented | 40 | --- | --- | --- | 41 | Policy as configuration | Store policies in YAML/JSON, not hardcoded. | Code deploy required for every policy change. | 42 | Most-restrictive-wins | Deny always overrides allow when composing policies. | A permissive policy bypasses a stricter one. | 43 | Pre-flight intent check | Classify intent before tool execution. | Dangerous action is detected only after side effects. | 44 | Tool allowlist | Permit only named tools for each role, context, or trust level. | Dynamic tool access expands beyond review. | 45 | Content filter | Block secrets, prompt injection, unsafe content, or regulated data flows. | Agent processes content it must not handle. | 46 | Human approval | Require approval for high-impact or irreversible operations. | Agent autonomously performs sensitive changes. | 47 | Rate limit | Bound requests, tool calls, tokens, or cost by actor and time window. | Runaway agent loops or cost spikes. | 48 | Trust decay | Decay trust scores over time. | Old good behavior grants permanent privilege. | 49 | Append-only audit | Never modify or delete audit entries. | Compliance evidence can be rewritten. | 50 | Fail closed | Deny action when governance checks error. | Safety system outage becomes allow-all. | 51 | Separate policy from logic | Keep enforcement independent from agent business logic. | Business code accidentally bypasses governance. | 52 53 ## Implementation checklist 54 55 ```markdown 56 ## Agent Governance Implementation Checklist 57 58 ### Setup 59 - [ ] Define governance policy (allowed tools, blocked patterns, rate limits) 60 - [ ] Choose governance level (open/standard/strict/locked) 61 - [ ] Set up audit trail storage 62 63 ### Implementation 64 - [ ] Add @govern decorator to all tool functions 65 - [ ] Add intent classification to user input processing 66 - [ ] Implement trust scoring for multi-agent interactions 67 - [ ] Wire up audit trail export 68 69 ### Validation 70 - [ ] Test that blocked tools are properly denied 71 - [ ] Test that content filters catch sensitive patterns 72 - [ ] Test rate limiting behavior 73 - [ ] Verify audit trail captures all events 74 - [ ] Test policy composition (most-restrictive-wins) 75 ``` 76 77 ## Framework integration cues 78 79 | Framework | Integration seam | 80 | --- | --- | 81 | PydanticAI | Wrap tools or dependencies with a policy check before invocation. | 82 | CrewAI | Gate crew tools and delegation tasks by role and trust level. | 83 | OpenAI Agents | Enforce tool allowlists and approval steps before tool calls execute. | 84 | LangChain | Wrap tools, callbacks, and chains with policy and audit middleware. | 85 | AutoGen | Gate inter-agent messages, delegation, and tool execution. | 86 87 Read `references/pattern-implementations.md` before writing code for policy objects, classifiers, decorators, trust scoring, audit trails, or framework-specific examples. 88 89 ## Criteria 90 91 - [ ] The agent's external tools, APIs, databases, file systems, and shell access are inventoried. 92 - [ ] Each tool has an allow, deny, approval, and audit rule. 93 - [ ] Intent classification runs before side-effecting tools. 94 - [ ] Dangerous prompt, sensitive data, and content filter rules are explicit. 95 - [ ] Trust scores have update and decay rules. 96 - [ ] Audit records are append-only and include actor, intent, tool, decision, timestamp, and outcome. 97 - [ ] Governance errors fail closed. 98 99 ## Gotchas 100 101 - **Do not hardcode policy in tool bodies**: policy must be reviewable and changeable outside business logic. 102 - **Do not rely on audit-only for production sensitive operations**: audit after harm is not prevention. 103 - **Do not let dynamic tool discovery bypass allowlists**: discovered tools still need explicit policy. 104 - **Do not delete audit entries**: corrections should append compensating records. 105 106 ## Progressive disclosure and bundled resources 107 108 - `references/pattern-implementations.md`: concrete implementations for policy, classifier, decorator, trust scoring, audit trail, and framework integration. 109 110 ## Environment shorthand 111 112 Use open governance for `dev/testing`; increase controls before production. 113 114 ## Output template 115 116 ```markdown 117 ## Agent governance plan 118 119 **Status:** ready | implemented | blocked 120 **Governance level:** open | standard | strict | locked 121 **Agent/framework:** <name> 122 123 | Control | Decision | Evidence or implementation | 124 | --- | --- | --- | 125 | Tool allowlist | <rule> | <policy path or code seam> | 126 | Intent classifier | <rule> | <classifier/prompt/model> | 127 | Human approval | <rule> | <approval path> | 128 | Audit trail | <rule> | <storage/export> | 129 | Trust scoring | <rule> | <update/decay formula> | 130 131 ### Validation 132 - Blocked tool denied: pass | fail 133 - Content filter tested: pass | fail 134 - Rate limit tested: pass | fail 135 - Audit event captured: pass | fail 136 ``` 137 138 ## Quality gate 139 140 - [ ] Governance level is selected based on risk: open, standard, strict, or locked. 141 - [ ] Policies are configuration or data, not scattered hardcoded conditionals. 142 - [ ] Most-restrictive-wins behavior is implemented or specified. 143 - [ ] Side-effecting tool calls require pre-flight intent and policy checks. 144 - [ ] Audit logging is append-only and records allowed and denied actions. 145 - [ ] Trust scoring includes decay over time. 146 - [ ] Fail-closed behavior is tested for policy, classifier, and audit failures. 147 148 ## References 149 150 - [Agent Governance Toolkit](https://github.com/microsoft/agent-governance-toolkit) 151 - [AgentMesh Integrations](https://github.com/microsoft/agent-governance-toolkit) 152 - [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/)
paulasilvatech