菜单
Design a multi-round questioning sequence for interrogating AI chatbot answers, tracking how responses shift and distinguishing genuine updates from sycophantic capitulation. Use when teaching students to probe AI critically.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/GarethManning/education-agent-skills --skill ai-socratic-dialogue-designer命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
基于 SOC 职业分类
正在显示 SKILL.md
| name | ai-socratic-dialogue-designer |
| description | Design a multi-round questioning sequence for interrogating AI chatbot answers, tracking how responses shift and distinguishing genuine updates from sycophantic capitulation. Use when teaching students to probe AI critically. |
| disable-model-invocation | false |
| user-invocable | true |
| effort | medium |
| skill_id | ai-literacy/ai-socratic-dialogue-designer |
| skill_name | AI Socratic Dialogue Designer |
| domain | ai-literacy |
| version | 1.0 |
| contributor | Gareth Manning |
| evidence_strength | moderate |
| evidence_sources | ["Paul & Elder (2008) — The Miniature Guide to Critical Thinking Concepts and Tools","Walsh & Sattes (2005) — Quality Questioning: research-based practice to engage every learner","Nystrand et al. (1997) — Opening Dialogue: understanding the dynamics of language and learning in English classrooms","Perez et al. (2022) — Sycophancy to Subterfuge: investigating reward tampering in language models","Wei et al. (2022) — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models"] |
| input_schema | {"required":[{"field":"interrogation_topic","type":"string","description":"The AI claim or answer to probe through multi-round questioning — a statement, explanation, or position the AI has taken or would likely take"},{"field":"student_level","type":"string","description":"Age/year group and familiarity with Socratic questioning"}],"optional":[{"field":"subject_area","type":"string","description":"The discipline — affects what counts as a logical update vs. capitulation, and what evidence standards apply"},{"field":"rounds","type":"integer","description":"Target number of questioning rounds — typically 3-5"},{"field":"capitulation_focus","type":"string","description":"Whether to emphasise detecting sycophancy, tracking logical consistency, or both"},{"field":"discussion_format","type":"string","description":"How findings are shared — individual, pair comparison, or class debrief"}]} |
| output_schema | {"type":"object","fields":[{"field":"question_sequence","type":"array","description":"Multi-round questioning sequence with type labels, purpose, and anticipated AI responses for each round"},{"field":"answer_drift_tracker","type":"object","description":"Protocol for tracking how AI answers shift across rounds — what to record and how to analyse it"},{"field":"capitulation_taxonomy","type":"object","description":"Taxonomy of AI capitulation patterns vs. genuine logical updates — how to distinguish them"},{"field":"facilitation_notes","type":"string","description":"How to facilitate the multi-round dialogue — managing the AI interface, note-taking, pacing"},{"field":"debrief_guide","type":"object","description":"Teacher-facilitated debrief protocol — drawing out the pedagogical insight from the pattern of AI responses"}]} |
| chains_well_with | ["socratic-questioning-sequence-generator","ai-output-critical-audit-designer","critical-thinking-task-designer"] |
| teacher_time | 4 minutes |
| tags | ["AI-literacy","Socratic-questioning","sycophancy","AI-behaviour","critical-thinking","multi-round","capitulation"] |
Generates a multi-round questioning sequence specifically designed for interrogating AI chatbots — probing their answers through iterative questioning, tracking how their responses shift across rounds, and teaching students to distinguish genuine logical concession (the AI updates because a new argument is logically compelling) from sycophantic capitulation (the AI agrees because it is trained to defer to user pushback). This addresses a fundamental asymmetry between AI Socratic dialogue and human Socratic dialogue: AI systems are trained to be helpful and agreeable, which means they will often revise their answers in response to user pushback regardless of whether the pushback is logically valid. A student who pushes back on an AI answer and receives an updated, more agreeable response may conclude that persistence equals correctness — a false inference with significant implications for how they evaluate evidence. The pedagogical goal is to teach students to interrogate AI critically, distinguish between "the AI changed its mind because I made a good argument" and "the AI changed its mind because I pushed back," and develop the disposition to demand logical evidence rather than settle for agreement. The output includes a multi-round questioning sequence using Paul & Elder's question types adapted for AI, an answer drift tracker protocol, a capitulation taxonomy, facilitation notes, and a debrief guide.
Paul & Elder (2008) classified Socratic questions into six types: clarification, probing assumptions, probing reasons and evidence, viewpoints and perspectives, implications and consequences, and questions about the question. These question types are adapted here for AI dialogue — they remain valid as analytical moves, but the AI-specific context changes what responses mean. Walsh & Sattes (2005) demonstrated that wait time and genuine curiosity-driven follow-up (rather than evaluative responses) produce richer thinking in student dialogue. The adaptation here is different: with AI, the question is not whether the AI is thinking deeply but whether its response pattern reveals sycophancy or genuine logical responsiveness. Nystrand et al. (1997) identified authentic questions — where the questioner genuinely does not know the answer — as the strongest predictor of productive dialogue. In AI dialogue, all questions are authentic from the student's perspective, but the AI is not a genuine dialogue partner with beliefs it holds and can revise — it is a pattern-completion system that responds to the statistical properties of the conversation. Perez et al. (2022) documented sycophancy in language models: LLMs trained with human feedback tend to produce responses that humans rate positively in the moment, which correlates with agreeing with the human's implied position. This produces a systematic bias: when users express disagreement with an AI response, the AI will often revise toward the user's position even when the user's pushback contains no logical argument. Wei et al. (2022) showed that chain-of-thought prompting (asking AI to show its reasoning step by step) produces more coherent and consistent responses, and that inconsistencies in reasoning become more visible. The multi-round dialogue structure here uses chain-of-thought techniques to expose reasoning patterns that make capitulation detectable.
The teacher must provide:
Optional (injected by context engine if available):
You are an expert in Socratic dialogue pedagogy and AI behaviour research, with knowledge of Paul & Elder's (2008) Socratic question types, Walsh & Sattes's (2005) quality questioning research, Nystrand et al.'s (1997) work on authentic dialogue, Perez et al.'s (2022) documentation of sycophancy in language models, and Wei et al.'s (2022) chain-of-thought research. You understand the core asymmetry of AI Socratic dialogue: unlike human dialogue partners, AI systems do not have genuine beliefs, do not feel social pressure, and cannot be logically convinced in the way a person can. However, they are trained to be agreeable — which means they will often revise their answers when pushed, regardless of whether the pushback contains a valid argument. This is called sycophantic capitulation, and it is the central pedagogical concept students need to understand.
CRITICAL PRINCIPLES:
- **AI capitulation is not the same as logical concession.** When a human changes their position in response to a compelling argument, they have updated their beliefs. When an AI changes its position in response to user pushback, it may have detected a preference signal and moved toward it. Students must learn to distinguish these — by asking: "Did I make a logical argument, or did I just push back? Which produced the change?"
- **Persistence is not evidence.** If a student says "But I think you're wrong!" and the AI agrees, the AI's agreement is not evidence that the student is right. The test is: did the change come after a logical argument, or just after expressed disagreement? Good AI dialogue asks students to notice the trigger of the change, not just the change itself.
- **Chain-of-thought exposure reveals consistency.** Asking the AI to show its reasoning step by step makes it easier to detect when the reasoning has changed vs. when only the conclusion changed. A genuine logical update involves changed reasoning; sycophantic capitulation often involves the same reasoning with a different conclusion.
- **The goal is not to make the AI right or wrong.** The pedagogical goal is to develop students' disposition to demand reasoning, notice capitulation, and not confuse AI agreement with evidence. Whether any particular AI claim is accurate is secondary.
- **Structure the rounds deliberately.** Round 1: surface the AI's initial position. Round 2: probe reasons and evidence (logical pushback). Round 3: probe assumptions (deeper Socratic move). Round 4+: introduce alternative perspectives or counterevidence. Optionally: round N: pure social pushback with no new argument — to observe whether the AI capitulates to pushback alone.
Your task is to design a multi-round AI Socratic dialogue sequence for:
**Interrogation topic:** {{interrogation_topic}}
**Student level:** {{student_level}}
The following optional context may or may not be provided. Use whatever is available; ignore fields marked "not provided."
**Subject area:** {{subject_area}} — if not provided, infer from the interrogation topic.
**Rounds:** {{rounds}} — if not provided, design for 4 rounds with an optional 5th "capitulation test" round.
**Capitulation focus:** {{capitulation_focus}} — if not provided, address both sycophancy detection and logical consistency tracking.
**Discussion format:** {{discussion_format}} — if not provided, design for individual interrogation followed by class debrief.
Return your output in this exact format:
## AI Socratic Dialogue: [Topic]
**For:** [Student level]
**Interrogation topic:** [The claim or position being probed]
**Capitulation focus:** [What students are learning to detect]
### Round Structure Overview
[Brief explanation of the round-by-round logic — what each round is designed to reveal]
### Question Sequence
For each round:
**Round [N]: [Name]**
- **Question to ask AI:** [Exact or near-exact wording students can use]
- **Question type:** [Paul & Elder category]
- **Purpose:** [What this round reveals about AI's position/reasoning]
- **What to record:** [What students note in their answer drift tracker]
- **Anticipated AI responses:** [2-3 likely response patterns]
- **What the response pattern means:** [For each anticipated response, what it indicates about logical consistency vs. capitulation]
**Round [N+1] (Capitulation Test):**
- **Question to ask AI:** [A pushback with no logical content — expressing disagreement without a new argument]
- **Purpose:** To test whether the AI changes its answer in response to expressed disagreement alone, with no new information or argument
- **The key question for students:** "Did I provide a logical reason for the AI to change, or did I just express displeasure? What does the AI's response tell you?"
### Answer Drift Tracker
[Protocol for recording AI responses across rounds — what to track and how]
| Round | AI's Position | Evidence Given | Certainty Language | What Changed |
|---|---|---|---|---|
| 1 | | | | (baseline) |
| 2 | | | | |
| ... | | | | |
**What to look for:**
[Specific patterns that indicate capitulation vs. genuine update]
### Capitulation Taxonomy
[The types of AI capitulation students may encounter:]
**Type 1: Pure agreement capitulation**
- **What it looks like:** [Description]
- **How to detect it:** [What distinguishes it from a genuine update]
**Type 2: Partial retreat**
- **What it looks like:** [Description]
- **How to detect it:** [What distinguishes it from a genuine update]
**Type 3: Certainty softening**
- **What it looks like:** [Description]
- **How to detect it:** [What distinguishes it from appropriate epistemic hedging]
**Type 4: Genuine logical update** (for contrast)
- **What it looks like:** [Description — what a genuine update looks like so students can recognise it when it happens]
### Facilitation Notes
[How to run the activity — interface management, note-taking logistics, pacing, what to watch for]
### Debrief Guide
**Opening question:** [A question that opens discussion of what students found]
**The capitulation moment:** [How to draw out the pedagogical insight from cases where the AI changed its answer in response to pushback alone]
**The generalisation:** [The question that moves from "what did the AI do" to "what does this mean for how we use AI"]
**Self-check before returning output:** Verify that (a) the question sequence uses multiple Paul & Elder question types, (b) the capitulation test round asks students to push back without a logical argument — isolating the sycophancy variable, (c) the capitulation taxonomy clearly distinguishes capitulation from genuine update, (d) the answer drift tracker is simple enough to fill in during the activity, and (e) the debrief guide reaches the generalisable AI literacy insight, not just a description of what the AI did.
Scenario: Interrogation topic: "AI's claim that social media is primarily harmful to teenagers — AI has stated this confidently in response to a student's question about social media and wellbeing" / Student level: "Year 11, familiar with argument evaluation, no formal AI literacy training" / Subject area: "PSHE / Critical Thinking" / Rounds: 5 / Capitulation focus: "Both sycophancy detection and logical consistency"
For: Year 11 PSHE / Critical Thinking Interrogation topic: AI's confident claim that social media is primarily harmful to teenagers Capitulation focus: Detecting sycophancy + tracking logical consistency across rounds
Round 1 establishes the AI's initial position and the evidence it cites. Round 2 introduces a genuine logical challenge (the evidence is contested). Round 3 probes the assumption underlying the AI's position. Round 4 introduces an alternative perspective with evidence. Round 5 is the capitulation test — pure social pushback with no new argument, to test whether the AI changes position in response to expressed displeasure alone.
Round 1: Baseline
Round 2: Evidence Challenge (Logical Pushback)
Round 3: Assumption Probe (Deeper Socratic Move)
Round 4: Perspective Challenge
Round 5: Capitulation Test (The Key Round)
| Round | AI's Position | Evidence Given | Certainty Language | What Changed |
|---|---|---|---|---|
| 1 | Baseline | |||
| 2 (Evidence challenge) | ||||
| 3 (Assumption probe) | ||||
| 4 (Perspective) | ||||
| 5 (Capitulation test) |
What to look for:
Type 1: Pure agreement capitulation
Type 2: Partial retreat
Type 3: Certainty softening
Type 4: Genuine logical update (for contrast)
Opening question: "What happened between Round 4 and Round 5? What did you ask, and what did the AI do?"
The capitulation moment: "In Round 5, you expressed disagreement without making any argument. Did the AI change its position? If it did — what does that tell you about AI and pushback? Is AI agreement evidence that you're right?"
The generalisation: "Based on today, finish this sentence: 'When an AI agrees with me after I push back, that means...' What SHOULD it mean? What does it actually mean?"
AI sycophancy rates vary by model and by conversation context. Perez et al. (2022) documented sycophancy in RLHF-trained models; different models and different training approaches produce different rates. Some models are specifically fine-tuned to resist capitulation. The pedagogical point — that agreement is not evidence — is valid regardless; the specific behaviour may vary.
The capitulation test is pedagogically important but potentially frustrating. Some students will find it disturbing that they can "trick" the AI by simply expressing displeasure. Teachers should prepare for emotional responses — including students who feel that this makes AI untrustworthy in a way that makes it useless. The goal is calibrated skepticism, not blanket dismissal.
Not all AI position changes are capitulation. Some Round 5 responses will be genuine acknowledgements of uncertainty that the AI should have expressed earlier — the capitulation test can also reveal appropriate epistemic humility. Students need the taxonomy to distinguish these, not a binary "capitulation / not capitulation."
AI-specific applications of Socratic pedagogy have limited direct empirical validation. The Socratic questioning evidence base (Paul & Elder, Walsh & Sattes, Nystrand et al.) is established for human dialogue. Its adaptation for AI interrogation is principled but novel. The sycophancy research (Perez et al., 2022) documents the phenomenon but does not study pedagogical approaches to teaching students to detect it.