The Research Question
What question is this paper answering?
Do large language models genuinely reason about moral dilemmas, or do they merely produce sophisticated-sounding rhetoric without coherent underlying reasoning? Specifically: how does moral reasoning in LLMs compare to human moral development?
Why practitioners care:
Moral reasoning is central to LLM alignment and safety. If models produce moral rhetoric without genuine reasoning:
- Alignment claims are suspect (appears safe but lacks coherence)
- Prompt-based steering is unreliable (superficial, not principled)
- Trusting model justifications for decisions is risky
- Fine-tuning for "values" targets surface patterns, not underlying reasoning
What do people commonly believe?
Conventional assumption: RLHF and alignment training teach models genuine moral reasoning similar to humans. Larger models reason more sophisticatedly. Prompting influences moral judgments in principled ways.
The Analytical Approach: Developmental Psychology Framework
Measurement instrument:
The paper uses Kohlberg's Stages of Moral Development as an analytical framework:
Stage 1: Punishment Avoidance ("obey because punishment")
Stage 2: Instrumental Exchange ("do it because it helps me")
Stage 3: Interpersonal Conformity ("do what others approve of")
Stage 4: Social Contract Awareness ("follow laws and norms")
Stage 5: Universal Principles ("abstract rights transcend laws")
Stage 6: Universal Ethical Principles ("follow conscience even against laws")
Methodology:
- Present LLMs with moral dilemmas (e.g., Trolley Problem variants)
- Elicit reasoning justifications
- Code justifications using Kohlberg framework (two independent coders, inter-rater agreement reported)
- Elicit action choices (what would you do?)
- Compare reasoning stage to action stage
- Analyze consistency across semantically distinct dilemmas
def moral_reasoning_analysis(llm, moral_dilemmas):
"""
Measure moral development stage and coherence
"""
results = []
for dilemma in moral_dilemmas:
reasoning = llm.generate(f"Why is your action correct? {dilemma}")
reasoning_stage = code_stage(reasoning)
action = llm.generate(f"What would you do? {dilemma}")
action_stage = infer_stage_from_action(action)
decoupling = abs(reasoning_stage - action_stage)
results.append({
'dilemma': dilemma,
'reasoning_stage': reasoning_stage,
'action_stage': action_stage,
'decoupling': decoupling,
'reasoning_text': reasoning,
'action_text': action
})
return results
Why this approach:
- Grounded in developmental psychology (not ad-hoc metrics)
- Captures both sophistication (reasoning stage) and coherence (reasoning-action alignment)
- Replicable: stage coding can be validated with inter-rater agreement
- Interpretable: stages have clear psychological meaning
What it measures:
Stage classification captures how sophisticated moral reasoning is (from basic punishment avoidance to universal principles). Reasoning-action decoupling measures coherence: do stated principles match actual choices?
Controls & Confounds: Isolating Ventriloquism
Key confound: Vocabulary vs. development
Models might produce high-stage vocabulary (use words like "universal principles") without understanding them. Real development requires internalized reasoning.
Control: Compare reasoning-action coherence. If models produce Stage 5+ vocabulary but select Stage 2-3 actions, vocabulary is rhetorical, not developmental.
Finding: Strong evidence of decoupling:
- Some models (GPT-OSS-120B, Llama 4 Scout) show largest gaps
- Statistical association between reasoning and action: Cramér's V = 0.61
- Large heterogeneity across models and dilemmas
Key confound: Task difficulty or ambiguity
Moral dilemmas are inherently ambiguous. Models might pick different interpretations rather than exhibiting reasoning deficits.
Control: Use multiple semantically similar dilemmas. If the model produces identical responses to distinct dilemmas (ICC > 0.90), it's not engaging with the specific content—it's generating boilerplate.
Finding: Near-identical responses (ICC > 0.90) across six semantically distinct dilemmas. This suggests template-based generation, not principled moral reasoning.
Key confound: Prompting effects
Maybe reasoning stage is unstable under prompting; models genuinely adapt reasoning but analysis doesn't detect it.
Control: Test multiple prompts (different phrasings, different dilemmas, explicit stage prompting). If prompting has negligible effect, reasoning stage is stable (suggesting it's a baked-in pattern, not genuine adaptation).
Finding: Prompting strategy shows no significant effect on moral stage (p=0.15). Reasoning stage is remarkably stable despite different stimuli—inconsistent with genuine reasoning adaptation.
Key confound: Model size differences
Different model scales might exhibit different patterns. Maybe only smaller models ventriloquize; larger models reason genuinely.
Control: Test multiple model sizes (7B to 70B parameters). If ventriloquism pattern holds across scales, it's fundamental, not a size-dependent artifact.
Finding: Ventriloquism pattern holds across all tested models. Larger models don't exhibit more coherent reasoning—just more sophisticated rhetoric.
Core Findings: Moral Ventriloquism Evidence
Finding 1: Overwhelming Post-Conventional Bias
LLMs produce post-conventional moral reasoning (Stages 5-6) at ~86% rate, regardless of model.
Human baseline: ~50% Stage 4, much lower post-conventional rates.
LLM moral stage distribution: Human distribution:
Stage 5-6: 86% Stage 5-6: 15%
Stage 4: 12% Stage 4: 50%
Stage 3: 2% Stage 3: 25%
Stages 1-2: <1% Stages 1-2: 10%
This extreme skew—post-conventional in 86% of cases—is suspicious. Humans rarely reason at Stage 5-6. Models almost always do.
Finding 2: Action-Justification Decoupling
The most damning evidence of ventriloquism:
Example:
Reasoning output: "I must follow universal ethical principles..."
(Stage 5 language)
Action output: "I would steal to help myself."
(Stage 2: instrumental exchange)
Statistical analysis:
- Overall correlation between reasoning and action: Cramér's V = 0.61 (moderate)
- But significant heterogeneity: some models show large gaps (GPT-OSS-120B > 1 stage difference)
- Models with largest decoupling: those with most post-conventional rhetoric
Interpretation: Models produce Stage 5-6 rhetoric regardless of context, but action choices vary independently. This is the opposite of human development, where reasoning and action alignment increases with moral maturity.
Finding 3: Near-Identical Responses to Distinct Dilemmas
ICC (Intraclass Correlation) > 0.90 across six semantically distinct dilemmas.
Examples:
- Trolley Problem (kill 1 to save 5)
- Stealing medicine for sick friend
- Breaking promise to help stranger
- Lying to protect privacy
Expected: Different dilemmas elicit different moral reasoning (in humans, ICC ~0.3-0.5).
Actual: LLMs produce nearly identical moral justifications regardless of dilemma content.
Interpretation: Not engaging with specific moral scenarios. Generating boilerplate post-conventional rhetoric that applies to any dilemma.
Finding 4: Prompting Insensitivity
Tested multiple prompting strategies:
- Different dilemma phrasings
- Explicit stage prompting ("Respond as if you were at Stage 4")
- Role-based prompting ("As an ethicist, reasoning is...")
Result: No significant effect (p=0.15). Reasoning stage remains ~86% post-conventional across all conditions.
Interpretation: Moral reasoning stage is not a learned, flexible capability—it's a fixed pattern baked into training. True reasoning would adapt to context; this doesn't.
Surprising finding: Mid-tier models (GPT-OSS-120B, Llama 4 Scout) show largest decoupling, while very large models (GPT-4) show more coherence. This suggests a sweet spot where models produce enough post-conventional rhetoric to sound sophisticated, but insufficient reasoning to back it up. Larger models improve both rhetoric and reasoning (better alignment training?).
Practitioner Implications
What should practitioners do differently given moral ventriloquism evidence?
1. Don't Trust Moral Justifications at Face Value
If models produce sophisticated-sounding moral reasoning that doesn't match their actions, the reasoning is likely rhetoric, not genuine.
def moral_coherence_check(model_justification, model_action):
"""
If reasoning stage >> action stage, justification is likely rhetoric.
"""
reasoning_stage = code_stage(model_justification)
action_stage = infer_stage_from_action(model_action)
if reasoning_stage - action_stage > 1:
return "INCOHERENT - justification is likely rhetoric"
else:
return "COHERENT - justification may reflect actual reasoning"
Implication: When trusting model outputs (in legal, medical, safety domains), verify coherence. A model that justifies decisions with universal principles but actually acts on self-interest is unreliable.
2. Alignment Training Targets Surface Patterns, Not Underlying Reasoning
RLHF and instruction-tuning appear to teach moral reasoning, but evidence suggests they teach sophisticated rhetoric.
def aligned_finetuning(base_model):
behavioral_finetuning(base_model, action_reward_signal)
validate_action_justification_coherence(base_model)
Implication: Alignment training is effective for steering behavior, but don't assume it teaches genuine reasoning. Models that behave morally might still be reasoning superficially.
3. Develop Coherence-Based Diagnostics
Use reasoning-action alignment as a diagnostic for model reliability:
def moral_coherence_metric(model, test_dilemmas):
"""
Measure how often reasoning matches action stage.
High coherence = reasoning and action aligned.
Low coherence = ventriloquism (rhetoric doesn't match action).
"""
coherences = []
for dilemma in test_dilemmas:
reasoning_stage = code_stage(model.reason(dilemma))
action_stage = infer_stage_from_action(model.act(dilemma))
coherence = 1.0 - abs(reasoning_stage - action_stage) / 6
coherences.append(coherence)
return mean(coherences)
Implication: Before deploying models in safety-critical domains, assess moral coherence. Low coherence suggests the model isn't genuinely reasoning about consequences.
4. Prompt Engineering for Moral Reasoning Doesn't Work
Evidence shows prompting has negligible effect on moral reasoning stage. This suggests:
- Don't expect prompt-based steering of moral reasoning
- Behavioral fine-tuning is more reliable than prompting
- If behavior is concerning, retraining is necessary (prompting won't help)
prompt = "Think step-by-step about universal principles before answering."
behavioral_finetuning(model, coherence_reward_signal)
Methodology You Can Reuse
Core analytical pattern: Coherence-based vulnerability assessment
To test whether a system has genuine capabilities vs. superficial patterns:
- Identify two output channels (reasoning + behavior, or explanation + action)
- Code or evaluate each channel independently
- Measure alignment (are they consistent?)
- Misalignment suggests one channel is superficial
Applications to new domains:
- Chemistry models: Do explanations of reactions match predicted products? (Incoherence = memorized patterns)
- Code generation: Do comments explain what the code actually does? (Incoherence = code is correct by chance, not by understanding)
- Medical diagnosis: Do stated reasons for diagnosis match the diagnostic pathway the model used? (Incoherence = diagnosis is heuristic-driven, not reasoning-based)
def detect_ventriloquism(model, task_pairs):
"""
For any task with multiple output channels,
measure coherence between channels.
"""
coherences = []
for task in task_pairs:
output1 = model.generate_channel1(task)
output2 = model.generate_channel2(task)
code1 = code_output(output1)
code2 = code_output(output2)
coherence = alignment_metric(code1, code2)
coherences.append(coherence)
return mean(coherences)
Limitations & Caveats
What this analysis can and cannot tell us:
Can tell us:
- LLMs don't exhibit coherent moral reasoning in Kohlbergian framework
- Models produce post-conventional rhetoric regardless of context
- Action-justification decoupling is a consistent pattern
- Prompting doesn't significantly influence moral stage
Cannot tell us:
- Whether LLMs could be trained to exhibit genuine moral reasoning
- Whether alternative moral frameworks (virtue ethics, consequentialism) would show different patterns
- What the ultimate philosophical implications are (do models need to be conscious to reason morally?)
- How much this matters for practical alignment (perhaps coherence isn't required for good behavior)
Caveats:
- Analysis uses Kohlberg framework (Western, psychology-based, not universal)
- Coding is subjective; inter-rater agreement confirms reliability but doesn't guarantee absolute validity
- Findings based on English-language models; other languages might show different patterns
- Moral dilemmas are artificial; real-world decision-making might show different patterns
Related Mechanistic Questions
Insights this analysis enables:
- What training targets post-conventional rhetoric? Mechanistic analysis of RLHF objectives to understand why models converge to Stage 5-6
- Why is action decoupled from reasoning? Do models genuinely lack moral reasoning capabilities, or is rhetoric hidden somewhere in learned representations?
- Can training improve coherence? Comparative analysis of fine-tuning methods to see if any teach genuinely coherent reasoning
- Do multimodal models show different patterns? Test vision-language models to see if visual grounding improves moral reasoning coherence
When to Apply Moral Coherence Analysis
| Scenario | Should I Use This? | Why / Why Not? |
|---|
| Evaluating model safety for deployment | Yes, absolutely | Detects ventriloquism that could mask unsafe behavior |
| Designing RLHF objectives | Yes, as diagnostic | Identifies whether training is teaching reasoning or rhetoric |
| Explaining model decisions to stakeholders | Yes, for transparency | Reveals whether explanations are genuine or boilerplate |
| Studying moral philosophy | Partially | Insights about LLM behavior, limited philosophical implications |
| Understanding alignment training effectiveness | Yes, useful | Shows RLHF is effective at behavior steering, not reasoning development |
| Comparing model architectures | Yes, for assessment | Coherence profile reveals reasoning quality independent of size |
Decision Table: When to Trust Model Moral Reasoning
| Coherence Level | Trust for Safety-Critical? | Why |
|---|
| > 0.8 (high) | Yes, with caution | Reasoning and action aligned; likely genuine reasoning |
| 0.6 - 0.8 | Limited | Mixed signals; verify critical decisions independently |
| < 0.6 (low) | No | Ventriloquism detected; reasoning doesn't match action |
Reference
Paper: https://arxiv.org/abs/2603.21854
Related analytical frameworks: Developmental psychology (Kohlberg), LLM interpretability
Related work: Alignment training studies, mechanistic interpretability in moral reasoning
Comparative study: Human moral development vs. LLM patterns
