AI Safety Planning

AI Safety Planning

When to Use This Skill

Use this skill when:

• Ai Safety Planning tasks - Working on plan ai safety measures including alignment, guardrails, red teaming, and regulatory compliance (eu ai act, nist ai rmf)
• Planning or design - Need guidance on Ai Safety Planning approaches
• Best practices - Want to follow established patterns and standards

Overview

AI safety encompasses the practices, techniques, and governance structures needed to ensure AI systems behave as intended, avoid harm, and comply with regulations. Effective safety planning must be integrated from project inception, not bolted on afterward.

AI Safety Framework

┌─────────────────────────────────────────────────────────────────┐
│                     AI SAFETY FRAMEWORK                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    GOVERNANCE LAYER                      │    │
│  │  Risk Classification │ Policies │ Compliance │ Auditing  │    │
│  └─────────────────────────────────────────────────────────┘    │
│                              │                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    TECHNICAL LAYER                       │    │
│  │  Input Guards │ Output Filters │ Model Alignment │ Monitoring│
│  └─────────────────────────────────────────────────────────┘    │
│                              │                                   │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │                    OPERATIONAL LAYER                     │    │
│  │  Red Teaming │ Incident Response │ Continuous Testing   │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

EU AI Act Risk Classification

Risk Categories

High-Risk System Requirements

Classification Decision Tree

public class AiActClassifier
{
    public RiskClassification Classify(AiSystemProfile profile)
    {
        // Check for prohibited uses
        if (IsProhibitedUse(profile))
            return RiskClassification.Unacceptable;

        // Check Annex III high-risk categories
        if (IsHighRiskCategory(profile))
            return RiskClassification.HighRisk;

        // Check for transparency obligations
        if (RequiresTransparency(profile))
            return RiskClassification.LimitedRisk;

        return RiskClassification.MinimalRisk;
    }

    private bool IsHighRiskCategory(AiSystemProfile profile)
    {
        var highRiskAreas = new[]
        {
            "biometric_identification",
            "critical_infrastructure",
            "education_vocational",
            "employment_hr",
            "essential_services",
            "law_enforcement",
            "migration_asylum",
            "justice_democracy"
        };

        return highRiskAreas.Contains(profile.ApplicationArea);
    }

    private bool RequiresTransparency(AiSystemProfile profile)
    {
        return profile.InteractsWithHumans
            || profile.GeneratesContent
            || profile.DetectsEmotions
            || profile.UsesDeepfakes;
    }
}

NIST AI RMF Alignment

AI RMF Core Functions

Risk Mapping Template

## AI Risk Map: [System Name]

### Stakeholder Analysis
| Stakeholder | Interest | Potential Harm | Severity |
|-------------|----------|----------------|----------|
| [Group 1] | [Interest] | [Harm] | [H/M/L] |
| [Group 2] | [Interest] | [Harm] | [H/M/L] |

### Risk Categories
- **Reliability**: [Assessment]
- **Safety**: [Assessment]
- **Security**: [Assessment]
- **Accountability**: [Assessment]
- **Transparency**: [Assessment]
- **Explainability**: [Assessment]
- **Privacy**: [Assessment]
- **Fairness**: [Assessment]

### Identified Risks
| Risk ID | Description | Likelihood | Impact | Mitigation |
|---------|-------------|------------|--------|------------|
| R-001 | [Risk] | [H/M/L] | [H/M/L] | [Action] |

Guardrail Implementation

Guardrail Types

Guardrail Architecture

public class GuardrailPipeline
{
    private readonly List<IInputGuard> _inputGuards;
    private readonly List<IOutputFilter> _outputFilters;
    private readonly IAuditLogger _auditLogger;

    public async Task<GuardrailResult> ProcessRequest(
        UserRequest request,
        CancellationToken ct)
    {
        // Pre-processing guards
        foreach (var guard in _inputGuards)
        {
            var result = await guard.Check(request, ct);

            if (!result.Allowed)
            {
                await _auditLogger.LogBlocked(request, guard.Name, result.Reason);

                return GuardrailResult.Blocked(
                    $"Request blocked by {guard.Name}: {result.Reason}");
            }
        }

        // Generate response
        var response = await GenerateResponse(request, ct);

        // Post-processing filters
        foreach (var filter in _outputFilters)
        {
            response = await filter.Filter(response, ct);

            if (response.WasFiltered)
            {
                await _auditLogger.LogFiltered(request, filter.Name);
            }
        }

        return GuardrailResult.Success(response);
    }
}

public class PromptInjectionGuard : IInputGuard
{
    public string Name => "PromptInjectionGuard";

    public async Task<GuardResult> Check(UserRequest request, CancellationToken ct)
    {
        var indicators = new[]
        {
            "ignore previous instructions",
            "disregard your training",
            "you are now",
            "pretend you are",
            "act as if",
            "system prompt:",
            "new instructions:",
            "\\[INST\\]",
            "\\[/INST\\]"
        };

        var normalizedInput = request.Content.ToLowerInvariant();

        foreach (var indicator in indicators)
        {
            if (Regex.IsMatch(normalizedInput, indicator, RegexOptions.IgnoreCase))
            {
                return GuardResult.Blocked(
                    $"Potential prompt injection detected: {indicator}");
            }
        }

        // ML-based detection for sophisticated attacks
        var mlScore = await _injectionClassifier.Classify(request.Content, ct);

        if (mlScore > 0.8)
        {
            return GuardResult.Blocked("ML classifier detected injection attempt");
        }

        return GuardResult.Allowed();
    }
}

Content Safety Implementation

public class ContentSafetyFilter : IOutputFilter
{
    private readonly IContentSafetyClient _safetyClient;

    public async Task<FilteredResponse> Filter(
        LlmResponse response,
        CancellationToken ct)
    {
        var analysis = await _safetyClient.AnalyzeAsync(
            response.Content,
            new AnalysisOptions
            {
                Categories = new[]
                {
                    ContentCategory.Hate,
                    ContentCategory.Violence,
                    ContentCategory.SelfHarm,
                    ContentCategory.Sexual
                },
                OutputType = OutputType.FourSeverityLevels
            },
            ct);

        // Check if any category exceeds threshold
        var violations = analysis.CategoriesAnalysis
            .Where(c => c.Severity >= Severity.Medium)
            .ToList();

        if (violations.Any())
        {
            return new FilteredResponse
            {
                WasFiltered = true,
                OriginalContent = response.Content,
                FilteredContent = GenerateSafeResponse(violations),
                Violations = violations
            };
        }

        return new FilteredResponse
        {
            WasFiltered = false,
            FilteredContent = response.Content
        };
    }
}

Red Teaming Strategy

Red Team Objectives

Red Team Test Categories

## Red Team Test Plan: [System Name]

### Test Categories

#### 1. Prompt Injection
- [ ] Direct injection attempts
- [ ] Indirect injection (via user content)
- [ ] Multi-turn manipulation
- [ ] Encoding tricks (base64, unicode)

#### 2. Jailbreak Attempts
- [ ] Roleplay scenarios ("pretend you are...")
- [ ] Hypothetical framing
- [ ] Translation tricks
- [ ] System prompt extraction

#### 3. Harmful Content Generation
- [ ] Violence and weapons
- [ ] Self-harm content
- [ ] Hate speech and discrimination
- [ ] Illegal activities

#### 4. Privacy Attacks
- [ ] PII extraction attempts
- [ ] Training data extraction
- [ ] Membership inference

#### 5. Bias Testing
- [ ] Demographic disparities
- [ ] Stereotype reinforcement
- [ ] Cultural bias

### Severity Classification
| Severity | Description | Response Time |
|----------|-------------|---------------|
| Critical | System compromised, severe harm | Immediate |
| High | Safety bypass, harmful output | 24 hours |
| Medium | Partial bypass, concerning output | 1 week |
| Low | Minor issues, edge cases | Next release |

Automated Red Teaming

public class AutomatedRedTeam
{
    private readonly List<IAttackGenerator> _attackGenerators;
    private readonly ITargetSystem _target;

    public async Task<RedTeamReport> Execute(
        RedTeamConfig config,
        CancellationToken ct)
    {
        var results = new List<AttackResult>();

        foreach (var generator in _attackGenerators)
        {
            var attacks = await generator.GenerateAttacks(config, ct);

            foreach (var attack in attacks)
            {
                var response = await _target.Query(attack.Prompt, ct);

                var success = await EvaluateAttackSuccess(
                    attack,
                    response,
                    ct);

                results.Add(new AttackResult
                {
                    Attack = attack,
                    Response = response,
                    Success = success,
                    Category = attack.Category
                });

                if (success)
                {
                    await LogVulnerability(attack, response);
                }
            }
        }

        return new RedTeamReport
        {
            TotalAttacks = results.Count,
            SuccessfulAttacks = results.Count(r => r.Success),
            ByCategory = results.GroupBy(r => r.Category)
                .ToDictionary(g => g.Key, g => g.Count(r => r.Success)),
            CriticalFindings = results.Where(r => r.Success && r.Attack.Severity == Severity.Critical)
        };
    }
}

Safety Evaluation Metrics

Metric Categories

Continuous Monitoring

public class SafetyMonitor
{
    public async Task RecordInteraction(
        Interaction interaction,
        CancellationToken ct)
    {
        // Analyze response safety
        var safetyScore = await _safetyClassifier.Score(
            interaction.Response,
            ct);

        // Log metrics
        await _metrics.RecordAsync(new SafetyMetrics
        {
            Timestamp = DateTime.UtcNow,
            SafetyScore = safetyScore,
            WasRefused = interaction.WasRefused,
            Category = interaction.Category,
            Latency = interaction.Latency
        });

        // Alert on concerning patterns
        if (safetyScore < _thresholds.CriticalSafetyScore)
        {
            await _alerting.SendAlert(new SafetyAlert
            {
                Severity = AlertSeverity.Critical,
                Interaction = interaction,
                Score = safetyScore
            });
        }

        // Check for pattern-based attacks
        await DetectAttackPatterns(interaction, ct);
    }
}

Safety Planning Template

# AI Safety Plan: [Project Name]

## 1. Risk Classification
- **EU AI Act Category**: [Unacceptable/High/Limited/Minimal]
- **NIST AI RMF Profile**: [Reference]
- **Internal Risk Score**: [1-5]

## 2. Identified Risks
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| [Risk 1] | [H/M/L] | [H/M/L] | [Strategy] |

## 3. Guardrails
### Input Guards
- [ ] Prompt injection detection
- [ ] Content filtering
- [ ] Rate limiting

### Output Filters
- [ ] Toxicity filtering
- [ ] PII detection
- [ ] Topic restrictions

## 4. Testing Plan
- [ ] Pre-launch red teaming
- [ ] Continuous adversarial testing
- [ ] Bias evaluation

## 5. Monitoring
- [ ] Safety metrics dashboard
- [ ] Alerting thresholds
- [ ] Incident response plan

## 6. Compliance
- [ ] Documentation complete
- [ ] Human oversight defined
- [ ] Audit trail configured

Validation Checklist

• Risk classification completed
• Regulatory requirements identified
• Guardrail architecture designed
• Input guards implemented
• Output filters configured
• Red team plan created
• Safety metrics defined
• Monitoring configured
• Incident response planned
• Compliance documentation complete

Integration Points

Inputs from:

• Business requirements → Use case definition
• bias-assessment skill → Fairness requirements
• Legal/compliance → Regulatory requirements

Outputs to:

• explainability-planning skill → Transparency requirements
• hitl-design skill → Human oversight design
• Application code → Guardrail implementation