Model Safety
Apply to AI features that process untrusted content, retrieve private data, call tools, make consequential recommendations, or act on external systems.
Review Areas
- Threat actors, misuse cases, prohibited outcomes, affected users, and consequence severity.
- Prompt injection and instruction/data separation across uploads, web content, retrieval, connectors, and tool outputs.
- Least-privilege tools, allowlists, argument validation, sandboxing, rate limits, approvals, and reversible actions.
- Data minimization, tenant isolation, retention, logging, redaction, model-provider boundaries, and secret handling.
- Hallucination, uncertainty, citations, abstention, validation, human review, and appeal or recovery paths.
- Adversarial evaluations, monitoring, incident response, and staged rollout.
Do not represent prompt wording alone as a security boundary. Recommend layered technical controls and state residual risk.
1---2name: model-safety3description: Review AI features for misuse, prompt injection, unsafe tool use, data exposure, unreliable outputs, oversight, and operational controls.4---56# Model Safety78Apply to AI features that process untrusted content, retrieve private data, call tools, make consequential recommendations, or act on external systems.910## Review Areas1112- Threat actors, misuse cases, prohibited outcomes, affected users, and consequence severity.13- Prompt injection and instruction/data separation across uploads, web content, retrieval, connectors, and tool outputs.14- Least-privilege tools, allowlists, argument validation, sandboxing, rate limits, approvals, and reversible actions.15- Data minimization, tenant isolation, retention, logging, redaction, model-provider boundaries, and secret handling.16- Hallucination, uncertainty, citations, abstention, validation, human review, and appeal or recovery paths.17- Adversarial evaluations, monitoring, incident response, and staged rollout.1819Do not represent prompt wording alone as a security boundary. Recommend layered technical controls and state residual risk.
egeetas