| name | sogptspotter-detecting-chatgpt-generated-answers |
| description | Detect AI-generated answers in Q&A content using Siamese embedding comparison with reference-answer anchoring. Trigger phrases: 'detect AI-generated answers', 'check if this answer is ChatGPT', 'flag LLM-generated Stack Overflow posts', 'is this answer written by AI', 'AI content detection for technical Q&A', 'screen answers for AI generation' |
SOGPTSpotter: Detecting AI-Generated Answers in Technical Q&A Content
This skill enables Claude to detect whether technical Q&A answers (Stack Overflow posts, forum replies, code review comments) were generated by an LLM rather than written by a human. It applies the SOGPTSpotter technique: instead of analyzing a suspect answer in isolation, you generate a controlled reference answer to the same question, then compare the suspect answer against both the reference and known-human answers using embedding similarity. This triangulation via anchor-positive-negative triplets is far more robust than single-text classifiers like GPTZero or DetectGPT, achieving 97.67% accuracy vs. ~80% for those tools.
When to Use
- When a user asks you to check whether a Stack Overflow answer, forum post, or technical explanation was likely written by ChatGPT or another LLM.
- When moderating a Q&A platform and need to triage a batch of answers for AI-generation signals.
- When a user pastes a suspicious answer and asks "did an AI write this?"
- When building or advising on an AI-content-detection pipeline for technical communities.
- When evaluating whether contributed code explanations, tutorials, or documentation show signs of LLM generation.
- When a user wants to understand the linguistic and structural markers that distinguish AI-generated technical writing from human-authored content.
Key Technique: Siamese Comparison with Reference Anchoring
Most AI-detection tools (GPTZero, DetectGPT, GLTR) analyze a single text in isolation, looking for statistical artifacts like token probability distributions or perplexity curves. SOGPTSpotter takes a fundamentally different approach: it generates a reference answer to the same question using a known LLM prompt, then measures how similar the suspect answer is to that reference versus to known-human answers. The intuition is that AI-generated answers to the same question will cluster together in embedding space, while human answers will be more diverse and idiosyncratic.
The architecture uses a Siamese Neural Network with two weight-sharing BigBird encoders. BigBird is chosen over BERT because its sparse attention mechanism handles long posts (>512 tokens) without truncation — critical for Stack Overflow answers that often include lengthy code blocks. During training, the network learns from triplets: an anchor (the reference answer), a positive (another AI-generated answer to the same question), and a negative (a human-written answer). The triplet loss with margin alpha=0.6 pushes AI-generated embeddings close to the reference and pulls human embeddings away.
At inference time, you don't need the trained model weights to apply the core insight. The practical technique is: (1) generate a reference answer to the same question, (2) compare the suspect answer to the reference on multiple linguistic dimensions, and (3) apply the known discriminative signals — AI answers show higher cosine similarity to references (>=0.5 threshold), more uniform sentence length, formulaic structure, and hedging language. The paper showed this approach beats RoBERTa by 1.88%, DetectGPT by 22.35%, and GPTZero by 21.71% on accuracy.
Step-by-Step Workflow
-
Extract the original question. Identify the exact question the suspect answer is responding to. If only the answer is provided, ask the user for the question or reconstruct it from context. The question is essential — detection accuracy depends on generating a comparable reference.
-
Generate a reference answer. Using Claude itself, produce a comprehensive, formal, verbose answer to the same question. This mirrors the paper's reference-answer generation (max 1000 tokens, designed to be thorough and structured). This reference serves as the anchor for comparison.
-
Perform structural comparison. Compare the suspect answer against the reference on these dimensions:
- Organizational similarity: Do both use the same heading structure, numbered steps, or bullet patterns?
- Opening/closing patterns: Does the suspect answer start with "To solve this..." or "Here's how you can..." (common LLM patterns)?
- Sentence length uniformity: Measure variance in sentence lengths. AI text tends toward uniform 15-25 word sentences; humans show higher variance.
- Hedging and filler: Count qualifying phrases ("It's worth noting", "Keep in mind", "However", "It's important to"). LLM answers use these at 3-5x the human rate.
-
Analyze lexical and stylistic signals. Check for these AI-generation markers identified in the paper's feature analysis:
- Vocabulary diversity (type-token ratio): AI answers tend toward slightly lower diversity with more repetitive phrasing.
- Excessive politeness and meta-commentary ("Great question!", "I hope this helps!", "Let me explain").
- Overly balanced presentation ("On one hand... on the other hand") even for questions with clear answers.
- Consistent use of markdown formatting (headers, bold, code blocks) even when the platform norms don't call for it.
-
Evaluate code-to-text ratio. The paper found that answers with extensive code and minimal text (<30 tokens of prose) are harder to classify. If the answer is >80% code, note reduced detection confidence. Focus analysis on the prose portions.
-
Check for adversarial evasion. The paper tested synonym substitution, perturbation, and paraphrasing attacks. Look for signs of light post-processing designed to evade detection:
- Inconsistent register (formal explanation suddenly using slang).
- Unusual synonym choices that don't match the author's apparent expertise level.
- Sentences that read awkwardly as if paraphrased from a smoother original.
Concrete Examples
Example 1: Analyzing a suspicious Stack Overflow answer
User: "Can you check if this answer about Python decorators was written by AI?"
Answer text: "To understand decorators in Python, let's break this down step by step.
A decorator is a function that takes another function as an argument and extends
its behavior without explicitly modifying it. Here's a simple example:
def my_decorator(func):
def wrapper():
print("Before function call")
func()
print("After function call")
return wrapper
It's worth noting that decorators are widely used in frameworks like Flask and
Django. The @syntax is syntactic sugar for applying the decorator. Keep in mind
that decorators can also accept arguments, which adds another layer of complexity.
I hope this helps clarify the concept!"
Approach:
- Generate a reference answer to "How do Python decorators work?"
- Compare structure: Both use step-by-step framing, code example, framework mentions — high overlap.
- Flag markers: "Let's break this down step by step" (LLM opener), "It's worth noting" (hedging), "Keep in mind" (hedging), "I hope this helps" (LLM closer), uniform sentence lengths (14-19 words in prose sections).
- Note code-to-text ratio is balanced (~40% code), so detection confidence is not reduced.
Output:
Classification: HIGH CONFIDENCE AI-GENERATED (score: 0.85)
Evidence:
- Structure: Follows the canonical LLM answer template — introduction, step-by-step,
code example, contextual note, closing pleasantry. Closely mirrors reference answer
structure.
- Hedging phrases (3 detected): "It's worth noting", "Keep in mind",
"adds another layer of complexity"
- LLM opening pattern: "let's break this down step by step"
- LLM closing pattern: "I hope this helps clarify the concept!"
- Sentence length variance: Low (std dev 2.8 words vs typical human 6-8)
- No personal experience, no opinionated recommendation, no SO-specific
conventions (no "Edit:", no @mentions, no "as of Python 3.x")
Caveats: The code itself is standard and not indicative either way.
Detection confidence is lower for the code portion.
Example 2: Analyzing a likely human-written answer
User: "Is this answer about Git rebasing AI-generated?"
Answer text: "Honestly rebase is one of those things that clicks after you mess it
up a few times lol. What I do is: git fetch origin, git rebase origin/main, fix
conflicts if any, git rebase --continue. If it goes sideways just git rebase --abort
and try again or use merge instead, nobody will judge you. The interactive rebase
(git rebase -i HEAD~3) is super useful for cleaning up commits before a PR though,
highly recommend getting comfortable with that once you've got basic rebase down."
Approach:
- Generate a reference answer to "How do I use Git rebase?"
- Compare structure: Reference is organized with headers and steps; suspect answer is a single informal paragraph — low overlap.
- Check markers: Colloquial language ("lol", "goes sideways", "nobody will judge you"), personal opinion ("highly recommend"), no hedging phrases, no meta-commentary.
Output:
Classification: LIKELY HUMAN-WRITTEN (score: 0.2)
Evidence:
- Structure: Single paragraph, no headers or formatting — diverges sharply
from typical LLM answer structure
- Informal register: "lol", "goes sideways", "nobody will judge you" —
inconsistent with LLM output patterns
- Personal experience marker: "What I do is..." with opinionated advice
- High sentence length variance (std dev 7.1 words)
- No hedging phrases, no meta-commentary, no polite closing
- Contains practical advice with personality ("once you've got basic
rebase down") typical of experienced SO users
Caveats: None significant. High confidence in human classification.
Example 3: Batch screening of forum answers
User: "I have 5 answers to screen for AI generation. Here they are: [answers 1-5]"
Approach:
- For each answer, identify or reconstruct the question it responds to.
- Generate one reference answer per question.
- Run the structural comparison and marker analysis on all five.
- Present a summary table with per-answer scores and top signals.
Output:
| # | Length | AI Score | Confidence | Top Signals |
|---|--------|----------|------------|--------------------------------------|
| 1 | 245 w | 0.82 | High | Hedging (4x), template structure |
| 2 | 89 w | 0.35 | Low | Informal tone, personal anecdote |
| 3 | 312 w | 0.71 | Moderate | Reference overlap, but has SO idioms |
| 4 | 28 w | 0.45 | Low* | *Too short for reliable detection |
| 5 | 198 w | 0.88 | High | "Let me explain", uniform length |
Recommendation: Answers #1 and #5 warrant manual review.
Answer #3 shows mixed signals — could be a human who writes formally.
Answer #4 is too short for reliable classification (paper threshold: >50 tokens).
Best Practices
- Do always generate a reference answer to the same question before comparing. This triangulation is the core advantage over single-text analysis and what separates this technique from naive keyword matching.
- Do weigh multiple signals together rather than relying on any single marker. A single "It's worth noting" doesn't make text AI-generated; four hedging phrases plus structural template-matching does.
- Do adjust confidence based on text length. The paper found detection is most reliable at 150+ tokens and degrades below 50 tokens. Always note when analyzing short text.
- Do consider the platform norms. A formal, well-structured answer on a corporate documentation site is normal; the same style on a casual forum is more suspicious.
- Avoid making binary declarations without evidence. Always present the specific textual markers that support your classification.
- Avoid treating code blocks as strong signals either way. Both humans and LLMs produce syntactically similar code. Focus analysis on the prose.
- Avoid assuming that good formatting alone indicates AI generation. Some experienced human contributors write well-formatted answers with headers and code blocks.
Error Handling
- Answer without a question: If the user provides only the answer text without the original question, attempt to infer the question from the answer content. Note that detection confidence is reduced without a proper reference comparison.
- Very short text (<30 tokens of prose): Warn that the paper found this is the primary failure mode (3 of 50 false negatives in the real-world study were <30 token answers). Report low confidence and suggest manual review.
- Code-dominant answers (>80% code): Flag that linguistic analysis applies only to the prose portions. If prose is minimal, the technique's reliability drops significantly.
- Multilingual content: The paper's dataset and evaluation are English-only. For non-English answers, note that the markers may not transfer and confidence should be reduced.
- Adversarially modified text: If the text shows signs of post-processing (inconsistent register, awkward synonym choices), note this as a potential evasion attempt. The paper showed SOGPTSpotter maintains 94.43-95.85% F1 under adversarial attacks, but without the trained model, heuristic detection of evasion is less reliable.
Limitations
- No trained model weights: This skill applies the paper's insights heuristically. The full SOGPTSpotter achieves 97.67% accuracy with a trained Siamese network; heuristic application of the same signals will be less precise.
- English technical Q&A only: The paper evaluated on English Stack Overflow content. Cross-language and non-technical domains (e.g., creative writing, legal text) were not tested.
- Evolving LLM outputs: The paper tested ChatGPT (GPT-3.5/4), LLaMA 3, Claude 3, and Gemini Flash. Newer models may produce text with fewer detectable artifacts. Cross-LLM degradation was 4.78-6.55% in the paper.
- Human-AI hybrid text: The technique assumes an answer is fully AI-generated or fully human-written. Answers where a human edits AI-generated text, or uses AI for parts, are harder to classify.
- Short answers: Below ~50 tokens of prose, there isn't enough signal for reliable detection. The paper's real-world case study confirmed this as the primary failure mode.
Reference
