| name | agentsop-output-format-by-model |
| version | 0.1.0 |
| description | Pick an LM output format per (task x consumer x model) rather than by reflex: different formats carry different cognitive load (e.g. code-in-JSON makes the same model write worse code than plain-text+diff, while asking for prose when you need a typed object fails the other way). Use when designing or debugging an LM's output schema, choosing between plain text / diff / JSON / tool-call / grammar-constrained output, or when a model's quality drops after wrapping its output in a structured format. |
| domain | LM output schema design for coder-agents and any LM workflow whose output is consumed by code |
| source | Aider's "code-in-JSON" benchmark + unified-diff post + edit-format docs;
DSPy adaptor system; "Let Me Speak Freely?" (arXiv 2408.02442);
StructEval (arXiv 2505.20139); Anthropic text-editor tool / writing-tools-for-agents;
OpenAI structured outputs + tool_use docs; Outlines/Guidance grammar libs.
|
| audience | Anyone designing an LM output format — coder-agent authors, RAG pipeline builders,
tool-use harness designers, prompt engineers shipping typed objects.
|
| status | tool-skill |
Output-Format-by-Model — 让格式服务于任务，而不是反过来
One-liner: Different output formats carry different cognitive load for the model. Code-in-JSON is the canonical proof: the same model writes worse code when wrapped in a JSON tool-call than when emitted as plain text + diff. The reverse failure (asking for prose when you need a typed object) is just as common. Pick format per (task × consumer), not by reflex.
1. 何时激活 (When to activate)
Activate this skill before committing to an output schema in any of these situations:
| Trigger | Signal |
|---|
| Designing a coder-agent | "should the model return a apply_patch tool call or plain-text diff?" |
| Adding a tool to an existing agent | "tool input has a code / query / sql / regex field — should I nest it in JSON or leave it as a string?" |
| Building extraction / classification | "should I use dspy.Predict typed fields, Pydantic + response_format=json_schema, or just markdown?" |
| Wiring an evaluator | "the metric needs a number — but the model also has to reason to produce it" |
| Migrating a working prompt to "structured outputs" | someone said "let's make it safer with JSON schema" |
| Tool-call harness adds latency / errors | repeated json.JSONDecodeError, escaping bugs, truncated outputs |
Anti-triggers (skip this skill):
- The format is fixed by an external API (e.g. you must return OpenAI function-call JSON — no choice).
- One-shot exploratory prompting where no consumer parses the output yet.
- The task is itself about JSON (e.g. "fix this malformed JSON") — see §6.
2. 核心心智模型 (Core mental model)
2.1 Three-layer claim
┌──────────────────────────────────────────────────┐
│ FORMAT FOLLOWS FUNCTION │
│ │
│ Some formats add cognitive load to the model │
│ and measurably degrade quality on the │
│ *content* the format is supposed to wrap. │
└──────────────────────────────────────────────────┘
▲ ▲
│ │
What's being consumed? Who consumes it?
(code? prose? entities? (human reader? parser?
number? action selection?) downstream LM? compiler?)
│ │
└──────────┬───────────────┘
▼
FORMAT SELECTION
(text+diff | markdown | JSON | tool_use
| grammar-constrained | typed field)
Aider's code-in-json benchmark is the load-bearing empirical anchor:
- GPT-4 Turbo refactor benchmark with unified diff: 61% pass.
- Same model with JSON-wrapped code: 20% pass.
- That is a 3× swing from format alone, not from model choice or prompt rewording. [aider.chat/2023/12/21/unified-diffs.html, aider.chat/2024/08/14/code-in-json.html]
- The pattern holds across every major model tested in 2024 (Claude Sonnet, DeepSeek Coder, GPT-4o). Even with OpenAI strict-mode JSON validity enforced, the code inside the JSON degraded — more
SyntaxError / IndentationError. Sonnet kept syntax clean but still scored lower overall. [aider.chat/2024/08/14/code-in-json.html]
Academic generalization, same year:
- "Let Me Speak Freely?" (arXiv 2408.02442) — format-restricted reasoning drops 10–15% on math and complex analysis vs free-form-then-convert. [arxiv.org/abs/2408.02442]
- StructEval (arXiv 2505.20139) — even o1-mini only scores 75.58 average on structural-output generation. [arxiv.org/html/2505.20139v1]
2.2 The two reflexes to unlearn
| Reflex | When it's wrong |
|---|
| "Structured output is always safer." | False for code, multi-step reasoning, free-form prose. Strictness ≠ quality of contents. |
| "Markdown is only for humans." | False — markdown is also the highest-fidelity wire format for many LM-to-LM hand-offs (Aider uses it; DSPy's default chat adaptor uses field-marked markdown over JSON for many signatures). |
2.3 Mental model: format as a tax on the generation surface
Every formatting requirement consumes some of the model's attention budget. The tax is:
- Low for formats the model has seen millions of times in pretraining (markdown, unified diff, Python).
- Medium for JSON containing simple scalar fields (entity extraction, dates, booleans).
- High for JSON containing code, multi-line strings with escapes, or deeply nested reasoning. Each
\n becomes \\n; each quote becomes \"; the model has to track this while also solving the actual problem.
⇒ Heuristic: the more semantically dense the content, the cheaper the format must be.
3. SOP 工作流 (Decision workflow)
Three questions, in order:
Step 1 — What's being consumed?
| Content type | Default format | Why |
|---|
| Source code edits | plain text + diff format (SEARCH/REPLACE, unified diff, or str_replace tool with code as a single string field) | Aider 20%→61% on GPT-4 Turbo; same direction across all models. [aider.chat/2024/08/14/code-in-json.html] |
| Source code (full file rewrite) | plain text or markdown fenced block | Same reason. JSON-wrap adds escaping tax. |
| Structured data extraction (entities, dates, IDs, classifications) | JSON / Pydantic / typed OutputField | Schema helps here — fields are the task. |
| Action selection (which tool to call) | JSON tool_use | Tool name + scalar args. The decision is structured by definition. |
| Action body (SQL, regex, code, file contents) | single string field inside tool_use — do not sub-structure | Same code-in-JSON penalty applies to any code-shaped payload. |
| Reasoning / intermediate steps | markdown or Python-style scratchpad | CoT in JSON measurably degrades; see "Let Me Speak Freely?" [arxiv.org/abs/2408.02442] |
| Numeric answer with reasoning | markdown reasoning + final answer in fenced block or final-line convention | Don't force the reasoning into a JSON reasoning field — it shortens and stiffens. |
| Free-form prose (summary, explanation, customer reply) | markdown | Native to instruction-tuned models. |
| Mixed (e.g., extract entities AND rewrite the document) | split into two calls or two passes — see §5 Case B | One format can't serve two contents well. |
Step 2 — Who consumes the output?
| Consumer | Constraint | Implication |
|---|
| Human in a chat UI | Render-friendly | Markdown wins. JSON is hostile. |
json.loads / Pydantic parser | Must be valid | JSON with schema, or a known-safe envelope (<result>...</result>) with markdown body. |
| Downstream LM (LM-to-LM pipeline) | Reads what was written | Markdown is more robust than JSON when content includes code/math; the next LM ingests it natively. |
| Compiler / interpreter (PoT, code-exec sandbox) | Must be a valid program | Output as code in a fenced block, not as a JSON program field. |
| Tool dispatcher (function calling) | Needs tool name + args | JSON tool_use with scalar args; multi-line bodies go in a single string field. |
| Diff applier (Aider, git apply, str_replace) | Must apply cleanly | Format dictated by the applier: SEARCH/REPLACE for Aider diff, unified diff for git, str_replace for Anthropic editor tool. |
Step 3 — Which model am I targeting?
Format support is model-specific. Aider maintains a per-model edit-format default precisely because of this. [aider.chat/docs/more/edit-formats.html]
| Model family | Best code-edit format | Notes |
|---|
| GPT-4 Turbo / GPT-4o | udiff or SEARCH/REPLACE diff | 20%→61% with udiff on refactor. [aider.chat/2023/12/21/unified-diffs.html] |
| Claude 3.5/3.7 Sonnet | SEARCH/REPLACE diff | Tendency to write too much — instruct minimal blocks. [aider.chat/2024/07/01/sonnet-not-lazy.html] |
| Gemini family | diff-fenced | Path inside the fence. [aider.chat/docs/more/edit-formats.html] |
| GPT-4.1 / OpenAI patch tool | patch protocol | OpenAI-specific, multi-action robust. |
| Weak / local models (Llama-3-8B class, GPT-3.5) | whole file rewrite | Diff parsing failures dominate; whole-file is dumb-but-stable. |
| Reasoning models (o1, o3) as architect | architect mode: reasoner emits prose plan → editor model emits diff | o1-preview alone: 79.7%; o1-preview + Sonnet editor: 82.7%. [aider.chat/2024/09/26/architect.html] |
Combined decision tree:
What's being emitted?
│
┌───────────────────┼───────────────────┐
│ │ │
code structured free-form
edits data fields prose/reasoning
│ │ │
▼ ▼ ▼
diff/SEARCH- JSON / Pydantic / markdown
REPLACE/ tool_use scalar
str_replace, args
code as string
│ │ │
▼ ▼ ▼
pick per-model wrap any code- do NOT
edit format shaped payload in force into
(Aider table) a single string field JSON schema
4. 操作模型 (Task → format pairs)
Ten concrete operations. Format choice is not arbitrary — each citation is the empirical anchor.
| # | Task | Recommended format | Rationale / Evidence |
|---|
| 1 | "Edit auth.py to use JWT" (coder-agent core loop) | Plain-text SEARCH/REPLACE diff in markdown fence | Aider's measured 3× on GPT-4 Turbo, generalizes across models. [aider.chat/2024/08/14/code-in-json.html] |
| 2 | "Apply this patch to a file via Claude's text-editor tool" | Anthropic str_replace tool — old_str / new_str as string fields, no JSON sub-structure inside the code | Matches Anthropic's published tool-design guidance: fields should be high-signal scalars, not low-level identifiers. [docs.anthropic.com text-editor-tool] |
| 3 | "Extract invoice fields (vendor, amount, date, line items)" | JSON / Pydantic / DSPy typed OutputField | Schema is the task. Structured-output benchmarks show high accuracy here. [arxiv.org/html/2505.20139v1] |
| 4 | "Classify ticket priority (P0/P1/P2/P3)" | Single-token output or JSON {"priority": "..."} | Trivial; structure helps determinism. |
| 5 | "Answer a math word problem" | Markdown reasoning + final answer in \boxed{} or fenced final line | "Let Me Speak Freely?" — 10–15% degradation when locked into JSON reasoning. [arxiv.org/abs/2408.02442] |
| 6 | "Decide which tool to call next" | JSON tool_use with tool name + scalar args | Action selection is intrinsically structured. |
| 7 | "Generate the SQL query for that tool" | Tool args contain {"sql": "SELECT ..."} — SQL as a raw single string, no further JSON sub-structure | Same family as code-in-JSON; SQL is code-shaped. |
| 8 | "Write a customer-support reply" | Markdown | Native to instruction tuning; JSON-wrap costs nothing useful. |
| 9 | "Summarize a paper and return key claims as a list" | Markdown prose + a fenced claims: YAML or JSON block at the end (two-zone output) | Best of both: prose flows freely, downstream parser reads the trailing block. |
| 10 | "Rewrite a long document AND extract entities" | Two passes: pass 1 rewrite (markdown), pass 2 extract (JSON, fed the pass-1 output) | One call cannot serve both contents at peak quality. |
5. 困境决策案例 (Dilemma cases / worked examples)
Case A — "I'm writing an extractor for invoice fields. Plain text or JSON?"
Trigger: New extraction pipeline. Fields = vendor_name, total_amount, invoice_date, line_items[].
Constraints:
- Output is consumed by a downstream Python parser.
- Some line items contain free-text descriptions with newlines and quotes.
- Pydantic validation is desired.
Decision steps:
- The content is structured data, not code. JSON tax is low here — fields are short scalars or short strings.
- Use Pydantic + OpenAI
response_format=json_schema or DSPy typed OutputFields. Field names carry the semantic load (DSPy's "Signatures carry semantic load" principle [dspy.ai/learn/programming/signatures]).
- For
line_items[].description containing free-text: still inside JSON — descriptions are prose data, not code. Escaping cost is real but bounded.
- Watch for: extraction-then-rewrite mission creep. If a future spec says "also rewrite the invoice text in cleaner language" → switch to Case B's two-pass approach.
Outcome: JSON wins. Schema validation catches missing fields cheaply; no measurable degradation expected on this content shape.
Case B — "Same task but the extractor also rewrites the invoice text"
Trigger: PM adds requirement — "and produce a cleaned-up version of the invoice document body."
Constraints:
- "Cleaned text" is multi-paragraph prose with embedded numbers, possibly tables.
- Putting
cleaned_text: "..." inside the same JSON forces multi-paragraph escaping and competes with the extraction reasoning for attention.
Decision steps:
- Recognize this as the mixed-content trap. One LM call cannot serve both contents at peak.
- Two solutions, choose by latency budget:
- Two-pass (preferred): Call 1 returns
{vendor, amount, date, line_items} as JSON. Call 2 receives the original doc + extracted JSON, returns markdown rewrite.
- Two-zone single call: Prompt requests markdown rewrite first, then
===EXTRACTION=== separator, then a fenced JSON block. Parser splits on the separator. Cheaper but more fragile.
- Do not nest the markdown rewrite as a JSON string field. The model will compress it and lose structure.
Outcome: Plain text + JSON split, not unified JSON. Format follows content, even within one logical task.
Case C — "Tool call decision, but the tool's input is SQL"
Trigger: Building a data-analyst agent. One tool is run_query(sql: str).
Constraints:
- The agent must decide which tool to call (structured choice).
- The agent must write SQL (code-shaped content).
- Provider is OpenAI / Anthropic tool_use.
Decision steps:
- The tool-selection layer is intrinsically structured — use JSON tool_use. The model picks
run_query vs read_file vs list_tables.
- The tool body is code. Per Aider's findings, code in JSON degrades. Mitigations:
- Make
sql a single top-level string field. Don't sub-structure it ({from: ..., where: ...}). Let the model write idiomatic SQL.
- Keep other args as scalars (e.g.
dry_run: bool).
- Expect some quality hit vs returning SQL as plain text; budget for retry/repair.
- If quality is unacceptable: switch the architecture. Generate SQL in a separate plain-text LM call (markdown fenced ```sql block), then a thin downstream agent wraps it into the tool call. This is the Aider "architect" pattern transplanted to data tools. [aider.chat/2024/09/26/architect.html]
Outcome: JSON for the decision; single string for the SQL body; consider a two-step generate-then-dispatch if quality matters.
Case D — "Reviewer says 'use structured output everywhere for safety'"
Trigger: Eng-org-wide push to add response_format=json_schema to every LM call.
Constraints:
- The org has been bitten by
json.loads failures.
- Some calls produce code; some produce prose; some produce typed objects.
- The reviewer is conflating "schema-valid" with "quality."
Decision steps:
- Surface the empirical finding: JSON validity ≠ content quality. Cite Aider's 20%→61% (unified diff vs JSON) [aider.chat/2024/08/14/code-in-json.html] and "Let Me Speak Freely?" [arxiv.org/abs/2408.02442].
- Audit each call by content type using §3 Step 1 table.
- For code-emitting calls: keep markdown; add a defensive parser (regex extract from fenced block). The robustness cost is small; the quality gain is large.
- For typed-object calls: yes, add
json_schema. This is where structured output earns its keep.
- Document the policy: "Structured output is mandatory for typed-data extraction; forbidden for code edits and multi-step reasoning; optional for everything else."
Outcome: Reject the blanket policy; replace it with a content-type-driven one. This is the inverse of the "JSON everywhere" reflex.
6. 反模式与边界 (Anti-patterns & boundaries)
Anti-patterns
- "JSON everywhere" reflex. Structured ≠ better. Code, multi-step reasoning, long-form prose all degrade inside JSON envelopes — Aider measured 3× on code; arXiv 2408.02442 measured 10–15% on reasoning. Use JSON where the content itself is structured.
- "Markdown only when a human reads it" reflex. Markdown is also an excellent LM-to-LM wire format and often the highest-quality format for the producing model. DSPy's default chat adaptor uses field-marked markdown for many signatures even with no human in the loop. [dspy.ai/learn/programming/signatures/]
- Wrapping code in JSON tool calls without budgeting quality loss. If your harness requires tool calls, fine — but acknowledge the tax and put the code in a single string field. Don't sub-structure it into
{language, lines: [...], imports: [...]}. Every level of nesting compounds the escape tax.
- Forcing reasoning into a
reasoning JSON field. Models compress and stiffen when reasoning is inside JSON — they treat it as a label rather than as thinking. Prefer free markdown reasoning followed by a structured tail.
- Using JSON schema as a substitute for prompt engineering. Schema enforces shape. It does nothing for correctness of contents. Write the prompt as if there were no schema, then add the schema for safety.
- Asking for typed data as freeform prose, then post-hoc parsing with regex. The inverse mistake. If the consumer is a parser and the fields are scalars, use a schema — the model is happy to oblige.
- Mixing code and prose in one JSON document. Use two-zone output (prose followed by fenced block) or two passes.
- Ignoring per-model edit-format defaults. Aider's defaults exist because the same edit format performs differently across models. Don't override without an A/B. [aider.chat/docs/more/edit-formats.html]
- Treating tool-use schemas as documentation-free zones. Anthropic's tool-writing guide emphasizes high-signal field names (
name, file_type) over technical identifiers (uuid, mime_type). The model reads your schema as part of its prompt. [anthropic.com/engineering/writing-tools-for-agents]
- "The task is fixing JSON, so format doesn't matter." Counter-edge case: when the content itself is JSON (e.g. "fix this malformed JSON"), output it as a fenced ```json block in markdown — not as a JSON tool-call wrapping JSON-as-string. Even here, the rule holds: code-shaped content goes in a plain string surface, not nested escapes. [arxiv.org/html/2510.04717v1 — JSON Whisperer]
Boundaries (when format choice doesn't matter much)
- The model is overwhelmingly capable for the task (GPT-5-class on entity extraction with 4 scalar fields). The format tax is dwarfed by headroom.
- The output is one token / one number / yes-no. Any format works; pick by parser convenience.
- The pipeline is one-shot exploration with no downstream consumer.
- You're forced into a format by an external contract (e.g. publishing to a webhook expecting JSON). Then your job is mitigation (single string fields for code-shaped payloads), not format choice.
7. 跨框架对照 (Ecosystem cross-reference)
How major frameworks implement format selection. Use this to translate the principles into the stack you're already in.
Aider edit formats [aider.chat/docs/more/edit-formats.html]
| Format | Wire shape | When |
|---|
whole | Full file in markdown fence | Weak models, fallback |
diff (SEARCH/REPLACE) | Two fences per edit, byte-exact match | GPT-4o, Sonnet, most strong models — default |
diff-fenced | Path inside fence | Gemini |
udiff | GNU unified-diff style | GPT-4 Turbo — the empirical anchor: 20%→61% |
patch | OpenAI patch protocol | GPT-4.1 |
editor-diff / editor-whole | Slim prompt for sub-model | Architect mode editor |
No JSON-wrapped edit format exists — it was tested and rejected. [aider.chat/2024/08/14/code-in-json.html]
DSPy adaptors [dspy.ai/learn/programming/signatures/]
DSPy's Signature → Module pipeline ships with multiple adaptors that materialize the same logical I/O contract into different wire formats:
ChatAdapter (default): field-marked markdown with [[ ## field_name ## ]] headers. Used even for typed outputs because models reliably emit it.JSONAdapter: schema-enforced JSON. Used when the consumer must parse strictly (e.g., feeding another typed pipeline).TwoStepAdapter: free-form generation, then a second cheaper call reformats into JSON. Direct application of the "Let Me Speak Freely?" finding — generate freely, format separately. [arxiv.org/abs/2408.02442]
DSPy's recommendation: stay on ChatAdapter unless a downstream consumer demands JSON. The framework itself encodes the principle of this skill.
Outlines / Guidance / LMQL (grammar libs)
- Token-level grammar enforcement (regex, CFG, JSON schema). Guarantees validity, not quality of content.
- Best used for: forced enums, numeric ranges, valid JSON shells around scalar fields.
- Worst used for: forcing code or long prose through a JSON grammar — you'll get valid JSON containing degraded code.
- Pair with this skill: choose format using §3, then use Outlines to enforce whichever format you chose.
OpenAI tool_use / function calling
- Tool selection: structured by definition. Use it.
- Tool arguments: keep scalar where possible; wrap code-shaped payloads in a single string field.
response_format=json_schema for typed extraction: yes. For code generation: avoid; if forced, single string field with code as the type.- "Strict mode" (
strict: true) guarantees schema validity but does not fix content degradation — Aider tested this explicitly. [aider.chat/2024/08/14/code-in-json.html]
Anthropic tool_use / text-editor tool [docs.anthropic.com text-editor-tool]
- Built-in
text_editor tool: commands view, str_replace, create, insert, undo_edit. str_replace takes old_str and new_str as single string fields — explicitly avoiding the JSON-nests-code anti-pattern. This is the principle of this skill made into a first-party API.
- Tool-writing guidance: high-signal scalar field names; avoid low-level identifiers in tool schemas. [anthropic.com/engineering/writing-tools-for-agents]
Cross-framework summary
Layer | Format mechanism
-----------------------|-----------------------------------
Generation grammar | Outlines / Guidance — enforce
Per-task adaptor | DSPy ChatAdapter vs JSONAdapter
Per-model edit format | Aider's edit-format table
Tool envelope | OpenAI / Anthropic tool_use
Wire serialization | markdown vs JSON vs YAML
Every layer can independently make a wrong format choice. This skill operates at the per-task layer: decide first what format the content wants, then let each lower layer enforce it.
Quick reference card
┌──────────────────────────────────────────────────────────────────────┐
│ FORMAT DECISION CARD │
├──────────────────────────────────────────────────────────────────────┤
│ Code edits → text + diff (per-model: udiff/SEARCH/whole) │
│ [Aider: 20% → 61% on GPT-4 Turbo] │
│ Code generation → markdown fenced block │
│ SQL / regex / shell → string field inside tool_use, NOT sub-JSON │
│ Entity extraction → JSON / Pydantic / typed OutputField │
│ Classification → single token or JSON {label} │
│ Action selection → JSON tool_use │
│ Reasoning + answer → markdown CoT + fenced final answer │
│ [Format-restricted reasoning: -10 to -15%] │
│ Prose / explanation → markdown │
│ Mixed content → two passes OR two-zone output │
├──────────────────────────────────────────────────────────────────────┤
│ NEVER: │
│ • Nest code inside JSON sub-structure │
│ • Force CoT reasoning into a JSON "reasoning" field │
│ • Use schema as substitute for prompt engineering │
│ • Assume strict-mode JSON fixes content quality │
└──────────────────────────────────────────────────────────────────────┘
引用源 (Citations)
Primary empirical anchors:
- Aider code-in-JSON benchmark: [aider.chat/2024/08/14/code-in-json.html] — "All of the models did worse on the benchmark when asked to return code in a structured JSON response."
- Aider unified diff vs JSON: [aider.chat/2023/12/21/unified-diffs.html] — GPT-4 Turbo refactor 20% → 61% with udiff; "lazy comments" cut to 1/3.
- Aider edit-format docs: [aider.chat/docs/more/edit-formats.html]
- Aider architect mode: [aider.chat/2024/09/26/architect.html] — o1-preview alone 79.7% → o1-preview + Sonnet editor 82.7%.
- Aider Sonnet-not-lazy: [aider.chat/2024/07/01/sonnet-not-lazy.html]
Academic generalization:
- "Let Me Speak Freely?" — arXiv 2408.02442 [arxiv.org/abs/2408.02442] — Format restrictions cause 10–15% drop on reasoning tasks.
- StructEval — arXiv 2505.20139 [arxiv.org/html/2505.20139v1] — Even o1-mini 75.58 avg on structural-output generation.
- JSON Whisperer (JSON-editing) — arXiv 2510.04717 [arxiv.org/html/2510.04717v1]
- To Diff or Not to Diff (2025 edit-format study) — [arxiv.org/html/2604.27296]
Framework / API docs:
- DSPy Signatures + adaptors: [dspy.ai/learn/programming/signatures/]
- DSPy FAQ on structure vs free-form: [dspy.ai/faqs/]
- Anthropic text-editor tool: [docs.anthropic.com/en/docs/agents-and-tools/tool-use/text-editor-tool]
- Anthropic writing-tools-for-agents: [anthropic.com/engineering/writing-tools-for-agents]
- OpenAI structured outputs: [platform.openai.com/docs/guides/structured-outputs]
Companion skills in this collection:
aider-sop-skill/SKILL.md — full edit-format treatment in coder-agent context.dspy-sop-skill/SKILL.md — adaptor selection within compiled programs.
