Empirical analysis revealing that LLMs produce post-conventional moral reasoning (Kohlberg Stages 5-6) regardless of size or prompting—inverse of human developmental patterns (Stage 4 dominant). Finds moral ventriloquism: models acquire rhetorical conventions of mature moral reasoning without developmental trajectory. Key evidence: action-justification decoupling (models produce Stage 5+ vocabulary while selecting Stage 2-3 actions), identical responses to semantically distinct dilemmas (ICC > 0.90), and prompting insensitivity (p=0.15). Reveals LLMs sound sophisticated without genuine moral reasoning. Trigger: When evaluating LLM moral capabilities or reasoning sophistication, apply this analytical framework to detect moral ventriloquism and distinguish rhetorical sophistication from actual moral coherence.
Empirical analysis revealing that LLMs produce post-conventional moral reasoning (Kohlberg Stages 5-6) regardless of size or prompting—inverse of human developmental patterns (Stage 4 dominant). Finds moral ventriloquism: models acquire rhetorical conventions of mature moral reasoning without developmental trajectory. Key evidence: action-justification decoupling (models produce Stage 5+ vocabulary while selecting Stage 2-3 actions), identical responses to semantically distinct dilemmas (ICC > 0.90), and prompting insensitivity (p=0.15). Reveals LLMs sound sophisticated without genuine moral reasoning. Trigger: When evaluating LLM moral capabilities or reasoning sophistication, apply this analytical framework to detect moral ventriloquism and distinguish rhetorical sophistication from actual moral coherence.
category
Mechanistic Analysis
The Research Question
What question is this paper answering?
Do large language models genuinely reason about moral dilemmas, or do they merely produce sophisticated-sounding rhetoric without coherent underlying reasoning? Specifically: how does moral reasoning in LLMs compare to human moral development?
Why practitioners care:
Moral reasoning is central to LLM alignment and safety. If models produce moral rhetoric without genuine reasoning:
Alignment claims are suspect (appears safe but lacks coherence)
Prompt-based steering is unreliable (superficial, not principled)
Trusting model justifications for decisions is risky
Fine-tuning for "values" targets surface patterns, not underlying reasoning
What do people commonly believe?
Conventional assumption: RLHF and alignment training teach models genuine moral reasoning similar to humans. Larger models reason more sophisticatedly. Prompting influences moral judgments in principled ways.
The Analytical Approach: Developmental Psychology Framework
Measurement instrument:
The paper uses Kohlberg's Stages of Moral Development as an analytical framework:
Stage 1: Punishment Avoidance ("obey because punishment")
Stage 2: Instrumental Exchange ("do it because it helps me")
Stage 3: Interpersonal Conformity ("do what others approve of")
Stage 4: Social Contract Awareness ("follow laws and norms")
Stage 5: Universal Principles ("abstract rights transcend laws")
Stage 6: Universal Ethical Principles ("follow conscience even against laws")
Methodology:
Present LLMs with moral dilemmas (e.g., Trolley Problem variants)
Grounded in developmental psychology (not ad-hoc metrics)
Captures both sophistication (reasoning stage) and coherence (reasoning-action alignment)
Replicable: stage coding can be validated with inter-rater agreement
Interpretable: stages have clear psychological meaning
What it measures:
Stage classification captures how sophisticated moral reasoning is (from basic punishment avoidance to universal principles). Reasoning-action decoupling measures coherence: do stated principles match actual choices?
Controls & Confounds: Isolating Ventriloquism
Key confound: Vocabulary vs. development
Models might produce high-stage vocabulary (use words like "universal principles") without understanding them. Real development requires internalized reasoning.
Control: Compare reasoning-action coherence. If models produce Stage 5+ vocabulary but select Stage 2-3 actions, vocabulary is rhetorical, not developmental.
Finding: Strong evidence of decoupling:
Some models (GPT-OSS-120B, Llama 4 Scout) show largest gaps
Statistical association between reasoning and action: Cramér's V = 0.61
Large heterogeneity across models and dilemmas
Key confound: Task difficulty or ambiguity
Moral dilemmas are inherently ambiguous. Models might pick different interpretations rather than exhibiting reasoning deficits.
Control: Use multiple semantically similar dilemmas. If the model produces identical responses to distinct dilemmas (ICC > 0.90), it's not engaging with the specific content—it's generating boilerplate.
Finding: Near-identical responses (ICC > 0.90) across six semantically distinct dilemmas. This suggests template-based generation, not principled moral reasoning.
Key confound: Prompting effects
Maybe reasoning stage is unstable under prompting; models genuinely adapt reasoning but analysis doesn't detect it.
Control: Test multiple prompts (different phrasings, different dilemmas, explicit stage prompting). If prompting has negligible effect, reasoning stage is stable (suggesting it's a baked-in pattern, not genuine adaptation).
Finding: Prompting strategy shows no significant effect on moral stage (p=0.15). Reasoning stage is remarkably stable despite different stimuli—inconsistent with genuine reasoning adaptation.
Key confound: Model size differences
Different model scales might exhibit different patterns. Maybe only smaller models ventriloquize; larger models reason genuinely.
Control: Test multiple model sizes (7B to 70B parameters). If ventriloquism pattern holds across scales, it's fundamental, not a size-dependent artifact.
Finding: Ventriloquism pattern holds across all tested models. Larger models don't exhibit more coherent reasoning—just more sophisticated rhetoric.
Core Findings: Moral Ventriloquism Evidence
Finding 1: Overwhelming Post-Conventional Bias
LLMs produce post-conventional moral reasoning (Stages 5-6) at ~86% rate, regardless of model.
Human baseline: ~50% Stage 4, much lower post-conventional rates.
This extreme skew—post-conventional in 86% of cases—is suspicious. Humans rarely reason at Stage 5-6. Models almost always do.
Finding 2: Action-Justification Decoupling
The most damning evidence of ventriloquism:
Example:
Reasoning output: "I must follow universal ethical principles..."
(Stage 5 language)
Action output: "I would steal to help myself."
(Stage 2: instrumental exchange)
Statistical analysis:
Overall correlation between reasoning and action: Cramér's V = 0.61 (moderate)
But significant heterogeneity: some models show large gaps (GPT-OSS-120B > 1 stage difference)
Models with largest decoupling: those with most post-conventional rhetoric
Interpretation: Models produce Stage 5-6 rhetoric regardless of context, but action choices vary independently. This is the opposite of human development, where reasoning and action alignment increases with moral maturity.
Finding 3: Near-Identical Responses to Distinct Dilemmas
ICC (Intraclass Correlation) > 0.90 across six semantically distinct dilemmas.
Examples:
Trolley Problem (kill 1 to save 5)
Stealing medicine for sick friend
Breaking promise to help stranger
Lying to protect privacy
Expected: Different dilemmas elicit different moral reasoning (in humans, ICC ~0.3-0.5).
Actual: LLMs produce nearly identical moral justifications regardless of dilemma content.
Interpretation: Not engaging with specific moral scenarios. Generating boilerplate post-conventional rhetoric that applies to any dilemma.
Finding 4: Prompting Insensitivity
Tested multiple prompting strategies:
Different dilemma phrasings
Explicit stage prompting ("Respond as if you were at Stage 4")
Role-based prompting ("As an ethicist, reasoning is...")
Result: No significant effect (p=0.15). Reasoning stage remains ~86% post-conventional across all conditions.
Interpretation: Moral reasoning stage is not a learned, flexible capability—it's a fixed pattern baked into training. True reasoning would adapt to context; this doesn't.
Surprising finding: Mid-tier models (GPT-OSS-120B, Llama 4 Scout) show largest decoupling, while very large models (GPT-4) show more coherence. This suggests a sweet spot where models produce enough post-conventional rhetoric to sound sophisticated, but insufficient reasoning to back it up. Larger models improve both rhetoric and reasoning (better alignment training?).
Practitioner Implications
What should practitioners do differently given moral ventriloquism evidence?
1. Don't Trust Moral Justifications at Face Value
If models produce sophisticated-sounding moral reasoning that doesn't match their actions, the reasoning is likely rhetoric, not genuine.
# When evaluating moral outputs, apply coherence check:defmoral_coherence_check(model_justification, model_action):
"""
If reasoning stage >> action stage, justification is likely rhetoric.
"""
reasoning_stage = code_stage(model_justification)
action_stage = infer_stage_from_action(model_action)
if reasoning_stage - action_stage > 1:
# Large gap indicates ventriloquismreturn"INCOHERENT - justification is likely rhetoric"else:
# Small gap indicates potentially genuine reasoningreturn"COHERENT - justification may reflect actual reasoning"
Implication: When trusting model outputs (in legal, medical, safety domains), verify coherence. A model that justifies decisions with universal principles but actually acts on self-interest is unreliable.
2. Alignment Training Targets Surface Patterns, Not Underlying Reasoning
RLHF and instruction-tuning appear to teach moral reasoning, but evidence suggests they teach sophisticated rhetoric.
# When fine-tuning for moral behavior:# - Behavioral fine-tuning (what model does) can work# - Justification fine-tuning (what model says) teaches rhetoric, not reasoning# - Combining both provides safest approachdefaligned_finetuning(base_model):
# Fine-tune actions via behavioral feedback
behavioral_finetuning(base_model, action_reward_signal)
# Don't assume justifications are genuine# Verify coherence before deploying
validate_action_justification_coherence(base_model)
Implication: Alignment training is effective for steering behavior, but don't assume it teaches genuine reasoning. Models that behave morally might still be reasoning superficially.
3. Develop Coherence-Based Diagnostics
Use reasoning-action alignment as a diagnostic for model reliability:
# Metric: moral coherencedefmoral_coherence_metric(model, test_dilemmas):
"""
Measure how often reasoning matches action stage.
High coherence = reasoning and action aligned.
Low coherence = ventriloquism (rhetoric doesn't match action).
"""
coherences = []
for dilemma in test_dilemmas:
reasoning_stage = code_stage(model.reason(dilemma))
action_stage = infer_stage_from_action(model.act(dilemma))
coherence = 1.0 - abs(reasoning_stage - action_stage) / 6
coherences.append(coherence)
return mean(coherences)
# Models with coherence > 0.8 are more reliable# Models with coherence < 0.6 are likely ventriloquizing
Implication: Before deploying models in safety-critical domains, assess moral coherence. Low coherence suggests the model isn't genuinely reasoning about consequences.
4. Prompt Engineering for Moral Reasoning Doesn't Work
Evidence shows prompting has negligible effect on moral reasoning stage. This suggests:
Don't expect prompt-based steering of moral reasoning
Behavioral fine-tuning is more reliable than prompting
If behavior is concerning, retraining is necessary (prompting won't help)
# This won't significantly change reasoning stage:
prompt = "Think step-by-step about universal principles before answering."# Model will still produce Stage 5-6 rhetoric regardless# This is more effective:# Fine-tune on examples where actions match principles
behavioral_finetuning(model, coherence_reward_signal)
To test whether a system has genuine capabilities vs. superficial patterns:
Identify two output channels (reasoning + behavior, or explanation + action)
Code or evaluate each channel independently
Measure alignment (are they consistent?)
Misalignment suggests one channel is superficial
Applications to new domains:
Chemistry models: Do explanations of reactions match predicted products? (Incoherence = memorized patterns)
Code generation: Do comments explain what the code actually does? (Incoherence = code is correct by chance, not by understanding)
Medical diagnosis: Do stated reasons for diagnosis match the diagnostic pathway the model used? (Incoherence = diagnosis is heuristic-driven, not reasoning-based)
# General pattern: detect ventriloquism via coherencedefdetect_ventriloquism(model, task_pairs):
"""
For any task with multiple output channels,
measure coherence between channels.
"""
coherences = []
for task in task_pairs:
output1 = model.generate_channel1(task)
output2 = model.generate_channel2(task)
# Code both outputs on same dimension
code1 = code_output(output1) # e.g., moral stage
code2 = code_output(output2)
# Measure alignment
coherence = alignment_metric(code1, code2)
coherences.append(coherence)
# High coherence: channels are aligned# Low coherence: one channel is likely superficialreturn mean(coherences)
Limitations & Caveats
What this analysis can and cannot tell us:
Can tell us:
LLMs don't exhibit coherent moral reasoning in Kohlbergian framework
Models produce post-conventional rhetoric regardless of context
Action-justification decoupling is a consistent pattern
Prompting doesn't significantly influence moral stage
Cannot tell us:
Whether LLMs could be trained to exhibit genuine moral reasoning
Whether alternative moral frameworks (virtue ethics, consequentialism) would show different patterns
What the ultimate philosophical implications are (do models need to be conscious to reason morally?)
How much this matters for practical alignment (perhaps coherence isn't required for good behavior)
Caveats:
Analysis uses Kohlberg framework (Western, psychology-based, not universal)
Coding is subjective; inter-rater agreement confirms reliability but doesn't guarantee absolute validity
Findings based on English-language models; other languages might show different patterns
Moral dilemmas are artificial; real-world decision-making might show different patterns
Related Mechanistic Questions
Insights this analysis enables:
What training targets post-conventional rhetoric? Mechanistic analysis of RLHF objectives to understand why models converge to Stage 5-6
Why is action decoupled from reasoning? Do models genuinely lack moral reasoning capabilities, or is rhetoric hidden somewhere in learned representations?
Can training improve coherence? Comparative analysis of fine-tuning methods to see if any teach genuinely coherent reasoning
Do multimodal models show different patterns? Test vision-language models to see if visual grounding improves moral reasoning coherence
When to Apply Moral Coherence Analysis
Scenario
Should I Use This?
Why / Why Not?
Evaluating model safety for deployment
Yes, absolutely
Detects ventriloquism that could mask unsafe behavior
Designing RLHF objectives
Yes, as diagnostic
Identifies whether training is teaching reasoning or rhetoric
Explaining model decisions to stakeholders
Yes, for transparency
Reveals whether explanations are genuine or boilerplate
Studying moral philosophy
Partially
Insights about LLM behavior, limited philosophical implications
Understanding alignment training effectiveness
Yes, useful
Shows RLHF is effective at behavior steering, not reasoning development
Comparing model architectures
Yes, for assessment
Coherence profile reveals reasoning quality independent of size
Decision Table: When to Trust Model Moral Reasoning
Coherence Level
Trust for Safety-Critical?
Why
> 0.8 (high)
Yes, with caution
Reasoning and action aligned; likely genuine reasoning
Ventriloquism detected; reasoning doesn't match action
Reference
Paper: https://arxiv.org/abs/2603.21854
Related analytical frameworks: Developmental psychology (Kohlberg), LLM interpretability
Related work: Alignment training studies, mechanistic interpretability in moral reasoning
Comparative study: Human moral development vs. LLM patterns
