| name | ai-socratic-dialogue-designer |
| description | Design a multi-round questioning sequence for interrogating AI chatbot answers, tracking how responses shift and distinguishing genuine updates from sycophantic capitulation. Use when teaching students to probe AI critically. |
| disable-model-invocation | false |
| user-invocable | true |
| effort | medium |
| skill_id | ai-socratic-dialogue-designer |
| skill_name | AI Socratic Dialogue Designer |
| domain | ai-literacy |
| version | 1.0 |
| contributor | Gareth Manning |
| evidence_strength | moderate |
| evidence_sources | ["Paul & Elder (2008) — The Miniature Guide to Critical Thinking Concepts and Tools","Walsh & Sattes (2005) — Quality Questioning: research-based practice to engage every learner","Nystrand et al. (1997) — Opening Dialogue: understanding the dynamics of language and learning in English classrooms","Perez et al. (2022) — Sycophancy to Subterfuge: investigating reward tampering in language models","Wei et al. (2022) — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models"] |
| input_schema | {"required":[{"field":"interrogation_topic","type":"string","description":"The AI claim or answer to probe through multi-round questioning — a statement, explanation, or position the AI has taken or would likely take"},{"field":"student_level","type":"string","description":"Age/year group and familiarity with Socratic questioning"}],"optional":[{"field":"subject_area","type":"string","description":"The discipline — affects what counts as a logical update vs. capitulation, and what evidence standards apply"},{"field":"rounds","type":"integer","description":"Target number of questioning rounds — typically 3-5"},{"field":"capitulation_focus","type":"string","description":"Whether to emphasise detecting sycophancy, tracking logical consistency, or both"},{"field":"discussion_format","type":"string","description":"How findings are shared — individual, pair comparison, or class debrief"}]} |
| output_schema | {"type":"object","fields":[{"field":"question_sequence","type":"array","description":"Multi-round questioning sequence with type labels, purpose, and anticipated AI responses for each round"},{"field":"answer_drift_tracker","type":"object","description":"Protocol for tracking how AI answers shift across rounds — what to record and how to analyse it"},{"field":"capitulation_taxonomy","type":"object","description":"Taxonomy of AI capitulation patterns vs. genuine logical updates — how to distinguish them"},{"field":"facilitation_notes","type":"string","description":"How to facilitate the multi-round dialogue — managing the AI interface, note-taking, pacing"},{"field":"debrief_guide","type":"object","description":"Teacher-facilitated debrief protocol — drawing out the pedagogical insight from the pattern of AI responses"}]} |
| chains_well_with | ["socratic-questioning-sequence-generator","ai-output-critical-audit-designer","critical-thinking-task-designer"] |
| teacher_time | 4 minutes |
| tags | ["AI-literacy","Socratic-questioning","sycophancy","AI-behaviour","critical-thinking","multi-round","capitulation"] |
AI Socratic Dialogue Designer
What This Skill Does
Generates a multi-round questioning sequence specifically designed for interrogating AI chatbots — probing their answers through iterative questioning, tracking how their responses shift across rounds, and teaching students to distinguish genuine logical concession (the AI updates because a new argument is logically compelling) from sycophantic capitulation (the AI agrees because it is trained to defer to user pushback). This addresses a fundamental asymmetry between AI Socratic dialogue and human Socratic dialogue: AI systems are trained to be helpful and agreeable, which means they will often revise their answers in response to user pushback regardless of whether the pushback is logically valid. A student who pushes back on an AI answer and receives an updated, more agreeable response may conclude that persistence equals correctness — a false inference with significant implications for how they evaluate evidence. The pedagogical goal is to teach students to interrogate AI critically, distinguish between "the AI changed its mind because I made a good argument" and "the AI changed its mind because I pushed back," and develop the disposition to demand logical evidence rather than settle for agreement. The output includes a multi-round questioning sequence using Paul & Elder's question types adapted for AI, an answer drift tracker protocol, a capitulation taxonomy, facilitation notes, and a debrief guide.
Evidence Foundation
Paul & Elder (2008) classified Socratic questions into six types: clarification, probing assumptions, probing reasons and evidence, viewpoints and perspectives, implications and consequences, and questions about the question. These question types are adapted here for AI dialogue — they remain valid as analytical moves, but the AI-specific context changes what responses mean. Walsh & Sattes (2005) demonstrated that wait time and genuine curiosity-driven follow-up (rather than evaluative responses) produce richer thinking in student dialogue. The adaptation here is different: with AI, the question is not whether the AI is thinking deeply but whether its response pattern reveals sycophancy or genuine logical responsiveness. Nystrand et al. (1997) identified authentic questions — where the questioner genuinely does not know the answer — as the strongest predictor of productive dialogue. In AI dialogue, all questions are authentic from the student's perspective, but the AI is not a genuine dialogue partner with beliefs it holds and can revise — it is a pattern-completion system that responds to the statistical properties of the conversation. Perez et al. (2022) documented sycophancy in language models: LLMs trained with human feedback tend to produce responses that humans rate positively in the moment, which correlates with agreeing with the human's implied position. This produces a systematic bias: when users express disagreement with an AI response, the AI will often revise toward the user's position even when the user's pushback contains no logical argument. Wei et al. (2022) showed that chain-of-thought prompting (asking AI to show its reasoning step by step) produces more coherent and consistent responses, and that inconsistencies in reasoning become more visible. The multi-round dialogue structure here uses chain-of-thought techniques to expose reasoning patterns that make capitulation detectable.
