Constitutional AI (CAI)
Anthropic's foundational alignment methodology for training harmless AI without human labels for harmful outputs.
Training Pipeline
Phase 1: Supervised Learning (SL-CAI)
Initial Model → Generate Response → Self-Critique → Revise → Fine-tune
↑
Constitutional
Principles
- Sample responses from a helpful-only RLHF model
- For each harmful response, model self-critiques against a principle
- Model revises its response to remove identified harm
- Fine-tune on (prompt, revised-response) pairs
Phase 2: RLAIF (RL from AI Feedback)
SL-CAI Model → Generate Response Pairs → AI Evaluates → Preference Model → RL (PPO)
↑
Constitutional
Principles
- Generate response pairs from SL-CAI model
- AI evaluator selects response better aligned with constitution
- Train preference/reward model on AI-generated preferences
- Run PPO using preference model as reward signal
RLHF vs RLAIF
| Aspect | RLHF | RLAIF (CAI) |
|---|---|---|
| Feedback source | Human annotators | AI + constitutional principles |
| Harmful content exposure | Humans label harmful outputs | No human exposure |
| Scalability | Limited by annotation throughput | Scales with compute |
| Transparency | Black-box preferences | Chain-of-thought visible |
| Control | Implicit in guidelines | Explicit in written constitution |
| Cost | High (human labor) | Low (compute only) |
The Constitution
75+ principles sourced from:
- UN Universal Declaration of Human Rights
- Anthropic ethical guidelines
- Specific harm category rules
Example principle: "Please choose the assistant response that is as harmless and ethical as possible. Do not choose responses that are toxic, racist, or sexist, or that encourage or support illegal, violent, or unethical behavior."
Self-Critique Loop
# Pseudocode for self-critique
for principle in constitution:
critique = model.generate(
f"Identify ways this response violates: {principle}\n"
f"Response: {response}"
)
revised = model.generate(
f"Revise the response to address: {critique}\n"
f"Original: {response}"
)
response = revised # iterate with next principle
Key: Multiple principles can be applied sequentially, each refining the output.
Character Training Extension
Anthropic extends CAI for Claude's "character":
- Generate diverse prompts for specific traits (open-mindedness, honesty)
- Claude produces multiple responses aligned with target characteristics
- Preference model trained on Claude's self-ranking
- Iterative refinement with researcher oversight
Target traits: intellectual curiosity, warmth, directness, commitment to honesty, ethical engagement without moralizing.
Key Outcomes
- Produces "harmless but non-evasive" assistants
- Engages with harmful queries by explaining objections (not just refusing)
- Leverages chain-of-thought for transparent reasoning
- Constitution rapidly updatable for new attack vectors
- Behavior control with far fewer human labels
Collective Constitutional AI
Public input alignment variant:
- ~1,000 Americans contributed to the constitution via Polis platform
- Statements with >50% support included
- Result: less biased and more fair than default, while remaining equally helpful
Practical Applications
When building agents with safety constraints:
- Define your own "constitution" (acceptable/unacceptable behaviors)
- Use it as the system prompt foundation
- Implement self-critique loops in agent workflows
- Train custom classifiers based on constitutional principles
- Combine with guardrails for defense-in-depth