# Constitutional AI

> Anthropic's Constitutional AI training methodology -- RLAIF, self-critique loops, character training, and alignment techniques for building harmless AI systems.

- Skill: `leolin990405/constitutional-ai` (Agent Skill)
- Install (CLI): `npx get-skill add leolin990405/constitutional-ai`
- Raw SKILL.md: https://api.skillmd.com/api/skills/leolin990405/constitutional-ai/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: LeoLin990405 (https://skillmd.com/u/leolin990405)
- Updated: 2026-11-11
- Page: https://skillmd.com/skills/leolin990405/constitutional-ai

---


# Constitutional AI (CAI)

> Anthropic's foundational alignment methodology for training harmless AI without human labels for harmful outputs.

## Training Pipeline

### Phase 1: Supervised Learning (SL-CAI)

```
Initial Model → Generate Response → Self-Critique → Revise → Fine-tune
                                         ↑
                                   Constitutional
                                    Principles
```

1. Sample responses from a helpful-only RLHF model
2. For each harmful response, model **self-critiques** against a principle
3. Model **revises** its response to remove identified harm
4. Fine-tune on (prompt, revised-response) pairs

### Phase 2: RLAIF (RL from AI Feedback)

```
SL-CAI Model → Generate Response Pairs → AI Evaluates → Preference Model → RL (PPO)
                                              ↑
                                        Constitutional
                                         Principles
```

1. Generate response pairs from SL-CAI model
2. AI evaluator selects response better aligned with constitution
3. Train preference/reward model on AI-generated preferences
4. Run PPO using preference model as reward signal

## RLHF vs RLAIF

| Aspect | RLHF | RLAIF (CAI) |
|--------|------|-------------|
| Feedback source | Human annotators | AI + constitutional principles |
| Harmful content exposure | Humans label harmful outputs | No human exposure |
| Scalability | Limited by annotation throughput | Scales with compute |
| Transparency | Black-box preferences | Chain-of-thought visible |
| Control | Implicit in guidelines | Explicit in written constitution |
| Cost | High (human labor) | Low (compute only) |

## The Constitution

75+ principles sourced from:
- UN Universal Declaration of Human Rights
- Anthropic ethical guidelines
- Specific harm category rules

Example principle: "Please choose the assistant response that is as harmless and ethical as possible. Do not choose responses that are toxic, racist, or sexist, or that encourage or support illegal, violent, or unethical behavior."

## Self-Critique Loop

```python
# Pseudocode for self-critique
for principle in constitution:
    critique = model.generate(
        f"Identify ways this response violates: {principle}\n"
        f"Response: {response}"
    )
    revised = model.generate(
        f"Revise the response to address: {critique}\n"
        f"Original: {response}"
    )
    response = revised  # iterate with next principle
```

Key: Multiple principles can be applied sequentially, each refining the output.

## Character Training Extension

Anthropic extends CAI for Claude's "character":
1. Generate diverse prompts for specific traits (open-mindedness, honesty)
2. Claude produces multiple responses aligned with target characteristics
3. Preference model trained on Claude's self-ranking
4. Iterative refinement with researcher oversight

Target traits: intellectual curiosity, warmth, directness, commitment to honesty, ethical engagement without moralizing.

## Key Outcomes

- Produces "harmless but non-evasive" assistants
- Engages with harmful queries by **explaining objections** (not just refusing)
- Leverages chain-of-thought for transparent reasoning
- Constitution rapidly updatable for new attack vectors
- Behavior control with far fewer human labels

## Collective Constitutional AI

Public input alignment variant:
- ~1,000 Americans contributed to the constitution via Polis platform
- Statements with >50% support included
- Result: less biased and more fair than default, while remaining equally helpful

## Practical Applications

When building agents with safety constraints:
1. Define your own "constitution" (acceptable/unacceptable behaviors)
2. Use it as the system prompt foundation
3. Implement self-critique loops in agent workflows
4. Train custom classifiers based on constitutional principles
5. Combine with guardrails for defense-in-depth

## Sources

- [Constitutional AI Paper](https://arxiv.org/abs/2212.08073)
- [Claude's Character](https://www.anthropic.com/research/claude-character)
- [Collective Constitutional AI](https://www.anthropic.com/research/collective-constitutional-ai-aligning-a-language-model-with-public-input)
- [Core Views on AI Safety](https://www.anthropic.com/news/core-views-on-ai-safety)

