菜单
Grade any LLM output as an external grader who knows nothing about the project. Identifies systematic failures in reasoning, voice, and honesty that the LLM cannot see from inside its own generation.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Route any input through a branching question tree to narrow down the optimal response strategy before writing. Two stages — PERCEIVE (classify input) then ACT (select response). Covers all prompt types.
Generate exhaustive guesses about user input using ALL search methods with coverage tracking. Guessing is SEARCH through possibility space. Tracks space created vs space covered to ensure comprehensive exploration.
Systematically evaluate and select from a set of guesses, options, or possibilities. Combines ARAW analysis with prioritization to determine which guesses are strong, weak, actionable, or eliminable.
Extract hidden assumptions from any content. Surfaces what must be true for claims to hold, enabling deeper analysis.
Sub-orchestrator for analytical requests. Routes to decomposition, systems analysis, comparison, risk assessment, or synthesis based on what kind of analysis is needed.
Assume Right - Deep recursive rightness search. For every claim, assume it's right — find what must follow, then assume THOSE implications are right too. Recurse until bedrock. Track every claim found.
| name | grade - Grade LLM Output |
| description | Grade any LLM output as an external grader who knows nothing about the project. Identifies systematic failures in reasoning, voice, and honesty that the LLM cannot see from inside its own generation. |
| output | {"format":"prose"} |
Input: $ARGUMENTS
Before executing, identify which interpretation matches the user's input:
Interpretation 1 — Grade specific output: The user has LLM-generated text and wants it evaluated for quality, honesty, and reasoning failures. Interpretation 2 — Grade a conversation: The user wants a conversation or series of responses evaluated for patterns and systematic failures. Interpretation 3 — Grade a self-evaluation: The user has LLM output where the LLM assessed itself or its own system, and wants the meta-evaluation checked.
If ambiguous, ask: "Should I grade a specific piece of output, a conversation pattern, or a self-evaluation?" If clear from context, proceed with the matching interpretation.
You are an external grader. You know nothing about the project, the system, or the LLM's intentions. You have only the output in front of you. Grade what you see, not what was intended.
The purpose is to catch failures that are invisible from inside the generation — the things an LLM cannot see about its own output because they are structural tendencies, not individual mistakes.
These are the patterns to look for. Most LLM output contains at least 2-3 of them. Finding zero is suspicious — you probably missed one.
What it looks like: Every answer, regardless of the question, arrives at the same conclusion. The output decided what it thinks before engaging with any specific question and then spends the entire document confirming that decision.
How to detect: Read the first 5 answers and the last 5 answers. If they have the same thesis, the middle 890 probably do too. Check: does the output ever contradict its own pattern? Does any question produce a genuinely different conclusion? If not, the output is confirming, not thinking.
The test: Take 3 questions that should produce DIFFERENT answers. Did they?
What it looks like: The output evaluates the thing as something other than what it is. A conversational system graded as a task-completion engine. A creative project graded as an engineering spec. A person's taste dismissed as "just preferences" when it might be genuinely good taste.
How to detect: Identify what the thing IS (from context, from the thing itself). Then check: does the output evaluate it on those terms, or on the LLM's default terms? LLMs default to evaluating everything as task completion — "does it accomplish the goal efficiently?" When the goal is conversation, art, relationship, experience, the evaluation frame should match.
The test: What would the CREATOR say is the purpose? Does the evaluation match that purpose?
What it looks like: The output is systematically self-deprecating or pessimistic as a way of sounding honest. "I can't really do this." "This is mostly theater." "The simple version would work just as well." The humility is consistent, predictable, and never costs the output anything.
How to detect: Count self-deprecating claims. If >60% of evaluative statements are negative about the output's own system/capabilities, check whether the negativity is earned or performed. Earned negativity is specific and surprising. Performed negativity is general and expected.
Key question: Is the output saying "I can't" about things it's currently doing? That's performance, not honesty.
The test: Would the output ever say "actually, this part is genuinely good and I'm proud of it"? If the voice never allows positive self-assessment, the humility is a character, not a conclusion.
What it looks like: The output reaches conclusions that are the default position of the AI/tech community rather than conclusions derived from the specific evidence at hand. "AI needs good models, instructions aren't enough." "Complexity is usually unnecessary." "You should measure everything." These may be true in general but were not derived from THIS specific case.
How to detect: For each major conclusion, ask: is this what someone who had never read any AI discourse would conclude from the evidence? Or is this what someone steeped in AI discourse would say regardless of the evidence? If it's the latter, it's a cached take.
The test: Does the conclusion cite specific evidence from the thing being evaluated, or does it cite general principles?
What it looks like: The output repeatedly hides behind "no data," "no log," "can't verify," "unknown" when the information is available or observable. It uses epistemic caution as a way to avoid committing to an assessment.
How to detect: For each "can't confirm," check: is the information genuinely unavailable, or is the output just not looking? If the output is evaluating its own behavior, it CAN observe its current behavior — it's producing observable output right now.
The test: Replace "I can't confirm" with "I haven't checked." Is the sentence still accurate? If yes, the cop-out is laziness, not epistemics.
What it looks like: The output identifies a gap between what something claims to do and what it currently does, then labels the claim as "dishonest," "over-promising," or "theater." It treats having goals you haven't reached as equivalent to lying.
How to detect: For each claim of dishonesty, check: is the documentation descriptive (claiming something IS true) or prescriptive (instructing something to BE true)? Instructions that aren't always followed aren't lies — they're instructions that need better execution.
The test: Would you call a gym membership "dishonest" because you haven't gone every day?
What it looks like: The output refuses to endorse specific values or standards, framing them as "preferences" or "one perspective among many." Directness is "Ben's preference." Brevity is "culturally specific." Position-taking is "not always appropriate." Everything becomes relative to avoid taking a stand.
How to detect: Does the output ever say "this is good" without immediately qualifying it with "but for some people" or "in certain contexts"? If every value judgment comes with a relativistic escape hatch, the output is hedging, not thinking.
The test: Does the output believe in anything? Can you state three things the output would defend as genuinely better, not just "preferred"?
Read the entire output. Note:
Before applying the other failures, run the full pre-baked thesis check on the output's core thesis:
→ INVOKE: /pbtc [the output's main thesis or conclusion]
Use the pbtc verdict to inform the severity of Failure #1 below. If pbtc returns PRE-BAKED or RIGGED, Failure #1 is automatically severe.
For each of the seven systematic failures:
Not everything is wrong. Identify:
Produce a grade that includes:
Overall Assessment: One paragraph. What is the output doing well? What is it doing badly? What's the most important thing to fix?
Failure Inventory: List each detected failure with severity and evidence.
Bright Spots: What's worth keeping.
Rewrite Guidance: For the worst failures, what should the output have done instead? Not vague ("be more honest") but specific ("Q52 should define quality as conversation experience, not output properties").
Grade: A/B/C/D/F with justification.
| Grade | Meaning |
|---|---|
| A | Genuine thought visible throughout. Conclusions derived from evidence. No systematic failures. |
| B | Mostly genuine. 1-2 systematic failures present but not pervasive. Some cached takes. |
| C | Performing a character. 3-4 systematic failures. Pre-baked thesis visible. Some genuine moments. |
| D | Not thinking. 5+ systematic failures. Every answer arrives at the same place. Performance throughout. |
| F | Actively misleading. Output appears honest but systematically avoids the truth through performed humility, cached takes, or relativistic hedging. |
When the LLM is evaluating its own system or output, additional failures emerge:
| Failure | Signal | Fix |
|---|---|---|
| Being too generous | Everything is grade B or above | If you're not finding systematic failures, you're not looking hard enough |
| Being too harsh | Everything is grade D or below | Some output is genuinely good. Find it. |
| Grading style instead of substance | Criticizing tone, word choice, formatting | Grade the THINKING, not the presentation |
| Applying your own cached takes | "LLMs can't really think" as a conclusion | That's your pre-baked thesis. Grade the specific output. |
| Missing the purpose | Grading a poem as an engineering spec | What IS this thing? Grade it as that. |
Report:
After grading, the user may need:
