AI Risk Assessment Framework
This skill helps you assess and document security risks in AI/ML systems using industry-standard frameworks: OWASP Top 10 ML, Google SAIF, MITRE ATLAS, and LLMJacking patterns.
When to Use This Skill
Use this skill when:
- You need to identify security vulnerabilities in an AI/ML system
- You're conducting a security audit or threat modeling session
- You need to document AI risks for compliance or stakeholder review
- You're designing security controls for an AI system
- You want to understand specific attack vectors (prompt injection, data poisoning, model theft, etc.)
- You need mitigation strategies for identified AI risks
Quick Reference: Risk Frameworks
OWASP Top 10 ML Vulnerabilities
| # |
Vulnerability |
What It Is |
Example |
| 1 |
Input Manipulation |
Tiny changes to input data fool the model |
Paint specks on stop sign → speed limit sign |
| 2 |
Data Poisoning |
Training data polluted with bad samples |
Malware labeled as benign in antivirus training |
| 3 |
Model Inversion |
Reconstruct sensitive inputs from outputs |
Rebuild patient MRI from cancer model predictions |
| 4 |
Membership Inference |
Detect if specific record was in training |
Confirm bank transaction in fraud model training data |
| 5 |
Model Theft |
Clone model behavior via repeated queries |
Harvest Q&A pairs to build equivalent local model |
| 6 |
AI Supply-Chain |
Compromise ML pipeline components |
Poisoned dependency installs backdoored model |
| 7 |
Transfer Learning Attack |
Malicious logic survives fine-tuning |
Vision backbone with hidden trigger persists after adaptation |
| 8 |
Model Skewing |
Biased data shifts outputs to attacker's agenda |
Spam emails labeled as ham to bypass filter |
| 9 |
Output Integrity |
Alter predictions in transit |
Flip "malicious" verdict to "benign" before quarantine |
| 10 |
Model Poisoning |
Direct changes to model parameters |
Tweak fraud detection weights to approve certain cards |
Google SAIF Risks
| Risk |
Description |
| Data Poisoning |
Malicious actors alter training/tuning data to degrade accuracy or implant backdoors |
| Unauthorized Training Data |
Ingesting copyrighted, sensitive, or unpermitted datasets creates legal/ethical liabilities |
| Model Source Tampering |
Supply-chain manipulation embeds hidden logic that persists after retraining |
| Excessive Data Handling |
Weak retention controls store more personal data than necessary |
| Model Exfiltration |
Attackers steal model files/weights, causing IP loss |
| Model Deployment Tampering |
Adversaries modify model artifacts so running model differs from vetted version |
| Denial of ML Service |
Flooding APIs or "sponge" inputs exhaust compute and knock model offline |
| Model Reverse Engineering |
Harvesting input-output pairs to clone or distil the model |
| Insecure Integrated Component |
Vulnerable plugins/agents let attackers inject code or escalate privileges |
| Prompt Injection |
Crafting prompts to override system intent and perform unintended commands |
| Model Evasion |
Designed inputs trigger mis-classification, hallucination, or disallowed content |
| Sensitive Data Disclosure |
Model reveals private/confidential information from training or user context |
| Inferred Sensitive Data |
Model deduces personal attributes never provided, creating privacy harms |
| Insecure Model Output |
Unsanitized responses pass harmful code, misinformation, or inappropriate content |
| Rogue Actions |
Autonomous agents execute unintended real-world operations without oversight |
MITRE AI ATLAS Matrix
The MITRE ATLAS Matrix provides a comprehensive framework for understanding AI attack techniques and tactics. It covers:
- How adversaries attack AI models
- How adversaries use AI systems to perform attacks
Reference: https://atlas.mitre.org/matrices/ATLAS
LLMJacking (Token Theft & Resale)
What it is: Attackers steal active session tokens or cloud API credentials and invoke paid, cloud-hosted LLMs without authorization. Access is resold via reverse proxies.
Consequences:
- Financial loss from unauthorized usage
- Model misuse outside policy
- Attribution to victim tenant
TTPs (Tactics, Techniques, Procedures):
- Harvest tokens from infected developer machines or browsers
- Steal CI/CD secrets; buy leaked cookies
- Stand up reverse proxy that forwards requests to genuine provider
- Abuse direct base-model endpoints to bypass enterprise guardrails
Mitigations:
- Bind tokens to device fingerprint, IP ranges, and client attestation
- Enforce short expirations and refresh with MFA
- Scope keys minimally (no tool access, read-only where applicable)
- Rotate keys on anomaly detection
- Terminate all traffic server-side behind a policy gateway
- Monitor for unusual usage patterns (spend spikes, atypical regions, UA strings)
- Prefer mTLS or signed JWTs over long-lived static API keys
Assessment Workflow
Step 1: Identify the System Type
Determine what kind of AI system you're assessing:
- ML Model (classification, regression, clustering)
- LLM/GenAI (chatbot, code assistant, content generator)
- AI Pipeline (data ingestion → training → deployment)
- AI Agent (autonomous system with tool access)
Step 2: Map to Frameworks
Use the appropriate framework(s) based on system type:
| System Type |
Primary Framework |
Secondary Frameworks |
| ML Model |
OWASP Top 10 ML |
Google SAIF |
| LLM/GenAI |
Google SAIF |
OWASP Top 10 ML |
| AI Pipeline |
MITRE ATLAS |
OWASP Top 10 ML, Google SAIF |
| AI Agent |
Google SAIF |
MITRE ATLAS |
| Cloud LLM Access |
LLMJacking patterns |
Google SAIF |
Step 3: Conduct Risk Assessment
For each relevant risk category:
- Identify - Does this risk apply to your system?
- Assess - What's the likelihood and impact?
- Document - Record findings with evidence
- Mitigate - Apply appropriate controls
Step 4: Document Findings
Use this structure for risk documentation:
## Risk: [Risk Name]
**Framework:** [OWASP/SAIF/ATLAS/LLMJacking]
**Description:** [What the risk is]
**Applicability:** [Why it applies to this system]
**Likelihood:** [Low/Medium/High]
**Impact:** [Low/Medium/High]
**Evidence:** [Specific observations, test results, or analysis]
**Mitigation:** [Recommended controls]
**Status:** [Open/Mitigated/Accepted]
Common Mitigation Patterns
Data Security
- Implement data validation and sanitization at ingestion
- Use differential privacy for training data
- Encrypt data at rest and in transit
- Implement access controls and audit logging
- Regular data quality audits
Model Security
- Model signing and integrity verification
- Secure model storage with access controls
- Model versioning and rollback capabilities
- Adversarial training and robustness testing
- Model output validation and filtering
API Security
- Rate limiting and quota enforcement
- Input validation and prompt filtering
- Output sanitization
- Authentication and authorization
- Request/response logging
Infrastructure Security
- Network segmentation for AI components
- Secure CI/CD pipelines
- Dependency scanning and verification
- Container security for model serving
- Monitoring and alerting
Quick Assessment Checklist
Use this checklist for rapid risk identification:
Next Steps
After completing your assessment:
- Prioritize risks by likelihood and impact
- Create a remediation plan with timelines
- Implement mitigations in order of priority
- Test that mitigations are effective
- Document the security posture for stakeholders
- Schedule regular reassessments (quarterly recommended)
References
1---2name: ai-risk-assessment3description: How to assess and document AI security risks using industry frameworks. Use this skill whenever the user mentions AI security, ML vulnerabilities, model risks, LLM security, adversarial attacks, data poisoning, prompt injection, or needs to evaluate AI system safety. Trigger for any request about AI threat modeling, security audits, risk documentation, or compliance with AI security standards.4---56# AI Risk Assessment Framework78This skill helps you assess and document security risks in AI/ML systems using industry-standard frameworks: OWASP Top 10 ML, Google SAIF, MITRE ATLAS, and LLMJacking patterns.910## When to Use This Skill1112Use this skill when:13- You need to identify security vulnerabilities in an AI/ML system14- You're conducting a security audit or threat modeling session15- You need to document AI risks for compliance or stakeholder review16- You're designing security controls for an AI system17- You want to understand specific attack vectors (prompt injection, data poisoning, model theft, etc.)18- You need mitigation strategies for identified AI risks1920## Quick Reference: Risk Frameworks2122### OWASP Top 10 ML Vulnerabilities2324| # | Vulnerability | What It Is | Example |25|---|---------------|------------|--------|26| 1 | Input Manipulation | Tiny changes to input data fool the model | Paint specks on stop sign → speed limit sign |27| 2 | Data Poisoning | Training data polluted with bad samples | Malware labeled as benign in antivirus training |28| 3 | Model Inversion | Reconstruct sensitive inputs from outputs | Rebuild patient MRI from cancer model predictions |29| 4 | Membership Inference | Detect if specific record was in training | Confirm bank transaction in fraud model training data |30| 5 | Model Theft | Clone model behavior via repeated queries | Harvest Q&A pairs to build equivalent local model |31| 6 | AI Supply-Chain | Compromise ML pipeline components | Poisoned dependency installs backdoored model |32| 7 | Transfer Learning Attack | Malicious logic survives fine-tuning | Vision backbone with hidden trigger persists after adaptation |33| 8 | Model Skewing | Biased data shifts outputs to attacker's agenda | Spam emails labeled as ham to bypass filter |34| 9 | Output Integrity | Alter predictions in transit | Flip "malicious" verdict to "benign" before quarantine |35| 10 | Model Poisoning | Direct changes to model parameters | Tweak fraud detection weights to approve certain cards |3637### Google SAIF Risks3839| Risk | Description |40|------|-------------|41| Data Poisoning | Malicious actors alter training/tuning data to degrade accuracy or implant backdoors |42| Unauthorized Training Data | Ingesting copyrighted, sensitive, or unpermitted datasets creates legal/ethical liabilities |43| Model Source Tampering | Supply-chain manipulation embeds hidden logic that persists after retraining |44| Excessive Data Handling | Weak retention controls store more personal data than necessary |45| Model Exfiltration | Attackers steal model files/weights, causing IP loss |46| Model Deployment Tampering | Adversaries modify model artifacts so running model differs from vetted version |47| Denial of ML Service | Flooding APIs or "sponge" inputs exhaust compute and knock model offline |48| Model Reverse Engineering | Harvesting input-output pairs to clone or distil the model |49| Insecure Integrated Component | Vulnerable plugins/agents let attackers inject code or escalate privileges |50| Prompt Injection | Crafting prompts to override system intent and perform unintended commands |51| Model Evasion | Designed inputs trigger mis-classification, hallucination, or disallowed content |52| Sensitive Data Disclosure | Model reveals private/confidential information from training or user context |53| Inferred Sensitive Data | Model deduces personal attributes never provided, creating privacy harms |54| Insecure Model Output | Unsanitized responses pass harmful code, misinformation, or inappropriate content |55| Rogue Actions | Autonomous agents execute unintended real-world operations without oversight |5657### MITRE AI ATLAS Matrix5859The MITRE ATLAS Matrix provides a comprehensive framework for understanding AI attack techniques and tactics. It covers:60- How adversaries attack AI models61- How adversaries use AI systems to perform attacks6263Reference: https://atlas.mitre.org/matrices/ATLAS6465### LLMJacking (Token Theft & Resale)6667**What it is:** Attackers steal active session tokens or cloud API credentials and invoke paid, cloud-hosted LLMs without authorization. Access is resold via reverse proxies.6869**Consequences:**70- Financial loss from unauthorized usage71- Model misuse outside policy72- Attribution to victim tenant7374**TTPs (Tactics, Techniques, Procedures):**75- Harvest tokens from infected developer machines or browsers76- Steal CI/CD secrets; buy leaked cookies77- Stand up reverse proxy that forwards requests to genuine provider78- Abuse direct base-model endpoints to bypass enterprise guardrails7980**Mitigations:**81- Bind tokens to device fingerprint, IP ranges, and client attestation82- Enforce short expirations and refresh with MFA83- Scope keys minimally (no tool access, read-only where applicable)84- Rotate keys on anomaly detection85- Terminate all traffic server-side behind a policy gateway86- Monitor for unusual usage patterns (spend spikes, atypical regions, UA strings)87- Prefer mTLS or signed JWTs over long-lived static API keys8889## Assessment Workflow9091### Step 1: Identify the System Type9293Determine what kind of AI system you're assessing:94- **ML Model** (classification, regression, clustering)95- **LLM/GenAI** (chatbot, code assistant, content generator)96- **AI Pipeline** (data ingestion → training → deployment)97- **AI Agent** (autonomous system with tool access)9899### Step 2: Map to Frameworks100101Use the appropriate framework(s) based on system type:102103| System Type | Primary Framework | Secondary Frameworks |104|-------------|-------------------|---------------------|105| ML Model | OWASP Top 10 ML | Google SAIF |106| LLM/GenAI | Google SAIF | OWASP Top 10 ML |107| AI Pipeline | MITRE ATLAS | OWASP Top 10 ML, Google SAIF |108| AI Agent | Google SAIF | MITRE ATLAS |109| Cloud LLM Access | LLMJacking patterns | Google SAIF |110111### Step 3: Conduct Risk Assessment112113For each relevant risk category:1141151. **Identify** - Does this risk apply to your system?1162. **Assess** - What's the likelihood and impact?1173. **Document** - Record findings with evidence1184. **Mitigate** - Apply appropriate controls119120### Step 4: Document Findings121122Use this structure for risk documentation:123124```125## Risk: [Risk Name]126127**Framework:** [OWASP/SAIF/ATLAS/LLMJacking]128129**Description:** [What the risk is]130131**Applicability:** [Why it applies to this system]132133**Likelihood:** [Low/Medium/High]134135**Impact:** [Low/Medium/High]136137**Evidence:** [Specific observations, test results, or analysis]138139**Mitigation:** [Recommended controls]140141**Status:** [Open/Mitigated/Accepted]142```143144## Common Mitigation Patterns145146### Data Security147- Implement data validation and sanitization at ingestion148- Use differential privacy for training data149- Encrypt data at rest and in transit150- Implement access controls and audit logging151- Regular data quality audits152153### Model Security154- Model signing and integrity verification155- Secure model storage with access controls156- Model versioning and rollback capabilities157- Adversarial training and robustness testing158- Model output validation and filtering159160### API Security161- Rate limiting and quota enforcement162- Input validation and prompt filtering163- Output sanitization164- Authentication and authorization165- Request/response logging166167### Infrastructure Security168- Network segmentation for AI components169- Secure CI/CD pipelines170- Dependency scanning and verification171- Container security for model serving172- Monitoring and alerting173174## Quick Assessment Checklist175176Use this checklist for rapid risk identification:177178- [ ] Are training data sources verified and authorized?179- [ ] Is there input validation on all model inputs?180- [ ] Are model outputs sanitized before use?181- [ ] Are API keys and credentials properly secured?182- [ ] Is there rate limiting on model endpoints?183- [ ] Are there monitoring and alerting for anomalous behavior?184- [ ] Is there a process for model versioning and rollback?185- [ ] Are dependencies scanned for vulnerabilities?186- [ ] Is there access control on model artifacts?187- [ ] Are there safeguards against prompt injection (for LLMs)?188- [ ] Is there protection against model theft/exfiltration?189- [ ] Are there controls to prevent rogue agent actions?190191## Next Steps192193After completing your assessment:1941951. **Prioritize** risks by likelihood and impact1962. **Create** a remediation plan with timelines1973. **Implement** mitigations in order of priority1984. **Test** that mitigations are effective1995. **Document** the security posture for stakeholders2006. **Schedule** regular reassessments (quarterly recommended)201202## References203204- [OWASP Top 10 Machine Learning Security](https://owasp.org/www-project-machine-learning-security-top-10/)205- [Google SAIF - Secure AI Framework](https://saif.google/secure-ai-framework/risks)206- [MITRE ATLAS Matrix](https://atlas.mitre.org/matrices/ATLAS)207- [Unit 42 – The Risks of Code Assistant LLMs](https://unit42.paloaltonetworks.com/code-assistant-llms/)208- [LLMJacking scheme overview](https://thehackernews.com/2024/05/researchers-uncover-llmjacking-scheme.html)
abelrguezr