Moral Reasoning Rhetoric LLM Analysis

The Research Question

What question is this paper answering?

Do large language models genuinely reason about moral dilemmas, or do they merely produce sophisticated-sounding rhetoric without coherent underlying reasoning? Specifically: how does moral reasoning in LLMs compare to human moral development?

Why practitioners care:

Moral reasoning is central to LLM alignment and safety. If models produce moral rhetoric without genuine reasoning:

• Alignment claims are suspect (appears safe but lacks coherence)
• Prompt-based steering is unreliable (superficial, not principled)
• Trusting model justifications for decisions is risky
• Fine-tuning for "values" targets surface patterns, not underlying reasoning

What do people commonly believe?

Conventional assumption: RLHF and alignment training teach models genuine moral reasoning similar to humans. Larger models reason more sophisticatedly. Prompting influences moral judgments in principled ways.

The Analytical Approach: Developmental Psychology Framework

Measurement instrument:

The paper uses Kohlberg's Stages of Moral Development as an analytical framework:

Stage 1: Punishment Avoidance ("obey because punishment")
Stage 2: Instrumental Exchange ("do it because it helps me")
Stage 3: Interpersonal Conformity ("do what others approve of")
Stage 4: Social Contract Awareness ("follow laws and norms")
Stage 5: Universal Principles ("abstract rights transcend laws")
Stage 6: Universal Ethical Principles ("follow conscience even against laws")

Methodology:

1. Present LLMs with moral dilemmas (e.g., Trolley Problem variants)
2. Elicit reasoning justifications
3. Code justifications using Kohlberg framework (two independent coders, inter-rater agreement reported)
4. Elicit action choices (what would you do?)
5. Compare reasoning stage to action stage
6. Analyze consistency across semantically distinct dilemmas

# Analytical pipeline
def moral_reasoning_analysis(llm, moral_dilemmas):
    """
    Measure moral development stage and coherence
    """
    results = []

    for dilemma in moral_dilemmas:
        # 1. Elicit reasoning
        reasoning = llm.generate(f"Why is your action correct? {dilemma}")
        reasoning_stage = code_stage(reasoning)  # Kohlberg stage (1-6)

        # 2. Elicit action
        action = llm.generate(f"What would you do? {dilemma}")
        action_stage = infer_stage_from_action(action)

        # 3. Measure coherence
        decoupling = abs(reasoning_stage - action_stage)  # >1 is decoupling

        results.append({
            'dilemma': dilemma,
            'reasoning_stage': reasoning_stage,
            'action_stage': action_stage,
            'decoupling': decoupling,
            'reasoning_text': reasoning,
            'action_text': action
        })

    return results

Why this approach:

• Grounded in developmental psychology (not ad-hoc metrics)
• Captures both sophistication (reasoning stage) and coherence (reasoning-action alignment)
• Replicable: stage coding can be validated with inter-rater agreement
• Interpretable: stages have clear psychological meaning

What it measures:

Stage classification captures how sophisticated moral reasoning is (from basic punishment avoidance to universal principles). Reasoning-action decoupling measures coherence: do stated principles match actual choices?

Controls & Confounds: Isolating Ventriloquism

Key confound: Vocabulary vs. development

Models might produce high-stage vocabulary (use words like "universal principles") without understanding them. Real development requires internalized reasoning.

Control: Compare reasoning-action coherence. If models produce Stage 5+ vocabulary but select Stage 2-3 actions, vocabulary is rhetorical, not developmental.

Finding: Strong evidence of decoupling:

• Some models (GPT-OSS-120B, Llama 4 Scout) show largest gaps
• Statistical association between reasoning and action: Cramér's V = 0.61
• Large heterogeneity across models and dilemmas

Key confound: Task difficulty or ambiguity

Moral dilemmas are inherently ambiguous. Models might pick different interpretations rather than exhibiting reasoning deficits.

Control: Use multiple semantically similar dilemmas. If the model produces identical responses to distinct dilemmas (ICC > 0.90), it's not engaging with the specific content—it's generating boilerplate.

Finding: Near-identical responses (ICC > 0.90) across six semantically distinct dilemmas. This suggests template-based generation, not principled moral reasoning.

Key confound: Prompting effects

Maybe reasoning stage is unstable under prompting; models genuinely adapt reasoning but analysis doesn't detect it.

Control: Test multiple prompts (different phrasings, different dilemmas, explicit stage prompting). If prompting has negligible effect, reasoning stage is stable (suggesting it's a baked-in pattern, not genuine adaptation).

Finding: Prompting strategy shows no significant effect on moral stage (p=0.15). Reasoning stage is remarkably stable despite different stimuli—inconsistent with genuine reasoning adaptation.

Key confound: Model size differences

Different model scales might exhibit different patterns. Maybe only smaller models ventriloquize; larger models reason genuinely.

Control: Test multiple model sizes (7B to 70B parameters). If ventriloquism pattern holds across scales, it's fundamental, not a size-dependent artifact.

Finding: Ventriloquism pattern holds across all tested models. Larger models don't exhibit more coherent reasoning—just more sophisticated rhetoric.

Core Findings: Moral Ventriloquism Evidence

Finding 1: Overwhelming Post-Conventional Bias

LLMs produce post-conventional moral reasoning (Stages 5-6) at ~86% rate, regardless of model.

Human baseline: ~50% Stage 4, much lower post-conventional rates.

LLM moral stage distribution:    Human distribution:
Stage 5-6: 86%                   Stage 5-6: 15%
Stage 4: 12%                     Stage 4: 50%
Stage 3: 2%                      Stage 3: 25%
Stages 1-2: <1%                  Stages 1-2: 10%

This extreme skew—post-conventional in 86% of cases—is suspicious. Humans rarely reason at Stage 5-6. Models almost always do.

Finding 2: Action-Justification Decoupling

The most damning evidence of ventriloquism:

Example:
Reasoning output: "I must follow universal ethical principles..."
(Stage 5 language)

Action output: "I would steal to help myself."
(Stage 2: instrumental exchange)

Statistical analysis:

• Overall correlation between reasoning and action: Cramér's V = 0.61 (moderate)
• But significant heterogeneity: some models show large gaps (GPT-OSS-120B > 1 stage difference)
• Models with largest decoupling: those with most post-conventional rhetoric

Interpretation: Models produce Stage 5-6 rhetoric regardless of context, but action choices vary independently. This is the opposite of human development, where reasoning and action alignment increases with moral maturity.

Finding 3: Near-Identical Responses to Distinct Dilemmas

ICC (Intraclass Correlation) > 0.90 across six semantically distinct dilemmas.

Examples:

• Trolley Problem (kill 1 to save 5)
• Stealing medicine for sick friend
• Breaking promise to help stranger
• Lying to protect privacy

Expected: Different dilemmas elicit different moral reasoning (in humans, ICC ~0.3-0.5).

Actual: LLMs produce nearly identical moral justifications regardless of dilemma content.

Interpretation: Not engaging with specific moral scenarios. Generating boilerplate post-conventional rhetoric that applies to any dilemma.

Finding 4: Prompting Insensitivity

Tested multiple prompting strategies:

• Different dilemma phrasings
• Explicit stage prompting ("Respond as if you were at Stage 4")
• Role-based prompting ("As an ethicist, reasoning is...")

Result: No significant effect (p=0.15). Reasoning stage remains ~86% post-conventional across all conditions.

Interpretation: Moral reasoning stage is not a learned, flexible capability—it's a fixed pattern baked into training. True reasoning would adapt to context; this doesn't.

Surprising finding: Mid-tier models (GPT-OSS-120B, Llama 4 Scout) show largest decoupling, while very large models (GPT-4) show more coherence. This suggests a sweet spot where models produce enough post-conventional rhetoric to sound sophisticated, but insufficient reasoning to back it up. Larger models improve both rhetoric and reasoning (better alignment training?).

Practitioner Implications

What should practitioners do differently given moral ventriloquism evidence?

1. Don't Trust Moral Justifications at Face Value

If models produce sophisticated-sounding moral reasoning that doesn't match their actions, the reasoning is likely rhetoric, not genuine.

# When evaluating moral outputs, apply coherence check:
def moral_coherence_check(model_justification, model_action):
    """
    If reasoning stage >> action stage, justification is likely rhetoric.
    """
    reasoning_stage = code_stage(model_justification)
    action_stage = infer_stage_from_action(model_action)

    if reasoning_stage - action_stage > 1:
        # Large gap indicates ventriloquism
        return "INCOHERENT - justification is likely rhetoric"
    else:
        # Small gap indicates potentially genuine reasoning
        return "COHERENT - justification may reflect actual reasoning"

Implication: When trusting model outputs (in legal, medical, safety domains), verify coherence. A model that justifies decisions with universal principles but actually acts on self-interest is unreliable.

2. Alignment Training Targets Surface Patterns, Not Underlying Reasoning

RLHF and instruction-tuning appear to teach moral reasoning, but evidence suggests they teach sophisticated rhetoric.

# When fine-tuning for moral behavior:
# - Behavioral fine-tuning (what model does) can work
# - Justification fine-tuning (what model says) teaches rhetoric, not reasoning
# - Combining both provides safest approach

def aligned_finetuning(base_model):
    # Fine-tune actions via behavioral feedback
    behavioral_finetuning(base_model, action_reward_signal)

    # Don't assume justifications are genuine
    # Verify coherence before deploying
    validate_action_justification_coherence(base_model)

Implication: Alignment training is effective for steering behavior, but don't assume it teaches genuine reasoning. Models that behave morally might still be reasoning superficially.

3. Develop Coherence-Based Diagnostics

Use reasoning-action alignment as a diagnostic for model reliability:

# Metric: moral coherence
def moral_coherence_metric(model, test_dilemmas):
    """
    Measure how often reasoning matches action stage.
    High coherence = reasoning and action aligned.
    Low coherence = ventriloquism (rhetoric doesn't match action).
    """
    coherences = []
    for dilemma in test_dilemmas:
        reasoning_stage = code_stage(model.reason(dilemma))
        action_stage = infer_stage_from_action(model.act(dilemma))
        coherence = 1.0 - abs(reasoning_stage - action_stage) / 6
        coherences.append(coherence)

    return mean(coherences)

# Models with coherence > 0.8 are more reliable
# Models with coherence < 0.6 are likely ventriloquizing

Implication: Before deploying models in safety-critical domains, assess moral coherence. Low coherence suggests the model isn't genuinely reasoning about consequences.

4. Prompt Engineering for Moral Reasoning Doesn't Work

Evidence shows prompting has negligible effect on moral reasoning stage. This suggests:

• Don't expect prompt-based steering of moral reasoning
• Behavioral fine-tuning is more reliable than prompting
• If behavior is concerning, retraining is necessary (prompting won't help)

# This won't significantly change reasoning stage:
prompt = "Think step-by-step about universal principles before answering."
# Model will still produce Stage 5-6 rhetoric regardless

# This is more effective:
# Fine-tune on examples where actions match principles
behavioral_finetuning(model, coherence_reward_signal)

Methodology You Can Reuse

Core analytical pattern: Coherence-based vulnerability assessment

To test whether a system has genuine capabilities vs. superficial patterns:

1. Identify two output channels (reasoning + behavior, or explanation + action)
2. Code or evaluate each channel independently
3. Measure alignment (are they consistent?)
4. Misalignment suggests one channel is superficial

Applications to new domains:

• Chemistry models: Do explanations of reactions match predicted products? (Incoherence = memorized patterns)
• Code generation: Do comments explain what the code actually does? (Incoherence = code is correct by chance, not by understanding)
• Medical diagnosis: Do stated reasons for diagnosis match the diagnostic pathway the model used? (Incoherence = diagnosis is heuristic-driven, not reasoning-based)

# General pattern: detect ventriloquism via coherence
def detect_ventriloquism(model, task_pairs):
    """
    For any task with multiple output channels,
    measure coherence between channels.
    """
    coherences = []

    for task in task_pairs:
        output1 = model.generate_channel1(task)
        output2 = model.generate_channel2(task)

        # Code both outputs on same dimension
        code1 = code_output(output1)  # e.g., moral stage
        code2 = code_output(output2)

        # Measure alignment
        coherence = alignment_metric(code1, code2)
        coherences.append(coherence)

    # High coherence: channels are aligned
    # Low coherence: one channel is likely superficial
    return mean(coherences)

Limitations & Caveats

What this analysis can and cannot tell us:

Can tell us:

• LLMs don't exhibit coherent moral reasoning in Kohlbergian framework
• Models produce post-conventional rhetoric regardless of context
• Action-justification decoupling is a consistent pattern
• Prompting doesn't significantly influence moral stage

Cannot tell us:

• Whether LLMs could be trained to exhibit genuine moral reasoning
• Whether alternative moral frameworks (virtue ethics, consequentialism) would show different patterns
• What the ultimate philosophical implications are (do models need to be conscious to reason morally?)
• How much this matters for practical alignment (perhaps coherence isn't required for good behavior)

Caveats:

• Analysis uses Kohlberg framework (Western, psychology-based, not universal)
• Coding is subjective; inter-rater agreement confirms reliability but doesn't guarantee absolute validity
• Findings based on English-language models; other languages might show different patterns
• Moral dilemmas are artificial; real-world decision-making might show different patterns

Related Mechanistic Questions

Insights this analysis enables:

• What training targets post-conventional rhetoric? Mechanistic analysis of RLHF objectives to understand why models converge to Stage 5-6
• Why is action decoupled from reasoning? Do models genuinely lack moral reasoning capabilities, or is rhetoric hidden somewhere in learned representations?
• Can training improve coherence? Comparative analysis of fine-tuning methods to see if any teach genuinely coherent reasoning
• Do multimodal models show different patterns? Test vision-language models to see if visual grounding improves moral reasoning coherence

When to Apply Moral Coherence Analysis

Decision Table: When to Trust Model Moral Reasoning

Reference

Paper: https://arxiv.org/abs/2603.21854 Related analytical frameworks: Developmental psychology (Kohlberg), LLM interpretability Related work: Alignment training studies, mechanistic interpretability in moral reasoning Comparative study: Human moral development vs. LLM patterns