Decision rubric for when an LM agent should write-and-run code (Program-of-Thought / code interpreter) versus reason in natural language: classify each step as deterministic- computable (emit + execute code, feed the result back) vs judgment (stay in prose). Use when designing or debugging an agent step that does arithmetic/parsing/data transforms, when prose reasoning hallucinates a computation (under-coding), or when a sandbox round- trip is wasted on a judgment task (over-coding). Search keywords: code interpreter, agent does math wrong, calculator hallucination, when to run code vs reason, program of thought, PoT, tool vs reasoning.
El comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Explorador de archivos
4 archivos
Mostrando SKILL.md
SKILL.md
Instrucciones de origen · Vista previa de solo lectura
name
agentsop-code-execution-decision
version
0.1.0
description
Decision rubric for when an LM agent should write-and-run code (Program-of-Thought / code interpreter) versus reason in natural language: classify each step as deterministic- computable (emit + execute code, feed the result back) vs judgment (stay in prose). Use when designing or debugging an agent step that does arithmetic/parsing/data transforms, when prose reasoning hallucinates a computation (under-coding), or when a sandbox round- trip is wasted on a judgment task (over-coding). Search keywords: code interpreter, agent does math wrong, calculator hallucination, when to run code vs reason, program of thought, PoT, tool vs reasoning.
domain
Deciding when an LM agent should write-and-run code (Program-of-Thought / code interpreter)
versus reason in natural language. An enhancement overlay on top of DSPy's ProgramOfThought
module, which ships the mechanism but not the decision rubric.
type
enhancement-overlay
source
DSPy ProgramOfThought module + docs [dspy.ai/learn/programming/modules/];
"Program of Thoughts" (Chen et al., arXiv 2211.12588);
PAL: Program-aided Language Models (Gao et al., arXiv 2211.10435);
OpenAI Code Interpreter / Assistants code-tool docs;
Anthropic code-execution tool docs;
LangChain PythonREPLTool;
sibling skill output-format-by-model (PoT for math/parse content shape).
audience
Coder-agent authors, tool-use harness designers, RAG/eval pipeline builders deciding
whether a step should emit executable code or stay in prose.
status
tool-skill
Code-Execution Decision — emit-code-vs-reason
One-liner: LMs are unreliable calculators but reliable coders. When the answer needs
determinism and precision — arithmetic, exact data manipulation, deterministic transforms —
emit code and run it. When the answer needs judgment, taste, or open-ended synthesis, reason
in natural language. The cost of getting this gate wrong is silent: prose arithmetic
hallucinates a plausible-looking wrong number, and over-coding a judgment task burns a
sandbox round-trip for nothing.
This is an enhancement overlay. DSPy already gives you dspy.ProgramOfThought (PoT) —
the mechanism for write-then-execute. What it does not give you is the decision rubric for
when to reach for it. That rubric is this skill. Cross-link the sibling
[[agentsop-output-format-by-model]] (which decides how code-shaped content should be serialized) and
[[agentsop-test-fix-loop]] (which closes the execute → error → retry loop).
1. 何时激活 (When to activate)
Activate this skill before committing a step to a reasoning strategy whenever the task has a
verifiable, deterministic core — or whenever you catch an agent doing arithmetic in prose.
Trigger
Signal
Arithmetic / math
"compute the compound interest", "what's 17.5% of $4,392.18", multi-step word problems, unit conversions, date deltas
Precise data manipulation
"sort these 240 rows by the third column", "dedupe and count", "join these two lists on id", "parse this CSV and sum column B"
Deterministic transforms
regex extraction, string reformatting, base conversion, hashing, sorting, set operations
Symbolic / combinatorial
"how many distinct permutations", "solve this system of equations", calendar/scheduling math
You see a model doing math in prose
"Let me add: 1,204 + 8,991 + ... = 10,195" — almost always worth a code check
Choosing a DSPy module
deciding between ChainOfThought and ProgramOfThought for a signature [dspy.ai/learn/programming/modules/]
Anti-triggers (do NOT reach for code execution):
The task is judgment: tone, summarization, ranking by quality, "is this reply empathetic?", design tradeoffs, open-ended explanation.
The "computation" is trivial and within the model's reliable range (single-digit arithmetic, a 3-item count) — the sandbox round-trip costs more than it saves.
No sandbox is available and the determinism requirement is soft.
The output's consumer is a human reading prose, and an approximate answer is acceptable.
2. 核心心智模型 (Core mental model)
LMs are unreliable calculators but reliable coders.
A language model predicts the next token, not the correct value. When you ask it to add
48,217 + 9,884 in prose, it emits the most plausible-looking digit sequence — which is
frequently wrong, and wrong in a way that looks right. The same model can write
48217 + 9884 as a Python expression flawlessly, because emitting the program is a
pattern-matching task it is genuinely good at, and the Python interpreter is a deterministic
oracle. This decoupling — model writes the recipe, interpreter computes the result — is the
entire thesis of Program-of-Thought (PoT) [arXiv 2211.12588] and PAL [arXiv 2211.10435].
┌────────────────────────────────────────────────────────┐
│ THE GATE: does this answer need determinism/precision? │
└────────────────────────────────────────────────────────┘
│ │
YES (computable) NO (judgment)
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ EMIT CODE │ │ REASON IN PROSE │
│ model writes recipe │ │ model is the engine │
│ interpreter = oracle│ │ no oracle exists │
└─────────────────────┘ └─────────────────────┘
│
▼
sandbox → run → feed result back into LM → LM narrates/uses it
Two failure modes the gate prevents:
Failure
Mechanism
Symptom
Under-coding (reason when you should compute)
LM hallucinates a calculation it cannot reliably perform
A confident, wrong, plausible-looking number; off-by-one counts; arithmetic that "looks" right
Over-coding (compute when you should reason)
LM wraps a judgment task in a sandbox round-trip that adds no determinism
Wasted latency + cost; brittle code that encodes a subjective rubric as if it were a formula; print("the tone is friendly")
Why the asymmetry matters. Under-coding fails silently — the wrong number propagates
downstream and no exception fires. Over-coding fails loudly and cheaply — you notice the
useless sandbox call. So the default lean, when genuinely uncertain and a verifiable core
exists, is toward code. But "genuinely uncertain" is the operative phrase: most judgment
tasks are not close calls.
The format corollary (from [[agentsop-output-format-by-model]]): once you decide to emit code, emit
it as code in a fenced block / single string field — never nested inside JSON sub-structure.
Code-in-JSON measurably degrades the code itself (Aider: 61%→20% on GPT-4 Turbo). The
execution decision and the serialization decision are two separate gates; pass both.
3. SOP 工作流 (Decision workflow)
A three-step gate. Run it per step, not per task — one task can have computable steps and
judgment steps interleaved.
Step 1 — Classify the step: deterministic-computable vs judgment
Ask: "Is there a single correct answer that a program could verify?"
Yes → computable. Arithmetic, sorting, parsing, counting, set/regex ops, symbolic math, anything with a checkable ground truth. → Step 2.
No → judgment. Quality ranking, tone, summarization, design choice, open synthesis. → reason in prose. Stop here.
Mixed → decompose. "Summarize these sales and tell me the total" = a judgment summary step + a computable total step. Route each independently. The total goes to code; the summary stays prose.
A useful tiebreaker for the "trivial computable" gray zone: if the model would be embarrassed
to get it wrong and you'd reach for a calculator yourself, emit code. If you'd do it in your
head without a second thought, prose is fine.
Step 2 — If computable, emit + run code
Emit the computation as a real program (Python is the PoT default; the interpreter is the oracle).
Keep the code minimal and side-effect-free for the computation — read inputs, compute, print/return the result. No network, no filesystem unless the task is I/O.
Serialize the code per [[agentsop-output-format-by-model]]: fenced block or single string field, not JSON-nested.
Step 3 — Sandbox → run → feed result back into the LM
Sandbox choice (see §4 op #3, cross-link [[agentsop-output-format-by-model]] for the wire shape): DSPy ProgramOfThought uses a Python interpreter (Deno/PythonInterpreter sandbox in recent versions); OpenAI Code Interpreter runs in a managed container; Anthropic code-execution tool runs in a sandboxed VM; LangChain PythonREPLTool runs in-process and is unsandboxed — treat as untrusted-input-hostile.
Run the code in the sandbox; capture stdout / return value / traceback.
Feed the result back into the LM — this is the step under-coders forget. The code produces a value; the LM must consume it to narrate, format, or chain into the next step. PoT is "code computes, LM contextualizes," not "code replaces the LM."
Error → retry (cross-link [[agentsop-test-fix-loop]]): on traceback, feed the error back to the LM, regenerate the code, re-run. Bound the retries (DSPy ProgramOfThought defaults to max_iters ≈ 3). After the bound, fall back to prose reasoning or surface the failure — do not loop forever.
Exit criterion: the step produces either (a) a code-derived value the LM has consumed, or
(b) a prose judgment, with the gate decision recorded so a reviewer can audit why code was or
wasn't used.
Inject stdout/return value into the next LM turn so the model narrates/uses it
LM-consumed result
PoT design: code computes, LM contextualizes [arXiv 2211.12588]
5
Error retry (bounded)
Code raised a traceback
Feed error to LM, regenerate, re-run; cap at max_iters (~3) then fall back
Fixed code or graceful fallback
DSPy ProgramOfThoughtmax_iters; see [[agentsop-test-fix-loop]]
6
Precision escalation
Prose answer involves multi-step arithmetic
Re-route the arithmetic to code even if prose started it
Code-verified number
"LMs are unreliable calculators" — PAL [arXiv 2211.10435]
7
Over-coding veto
About to sandbox a judgment task
Stop: no deterministic core → reasoning, not code
Prose reasoning, no sandbox call
§5 Case B; avoids wasted round-trip
In DSPy terms, op #1 is exactly the choice between dspy.ChainOfThought (prose reasoning) and
dspy.ProgramOfThought (emit+run) for a signature — the dspy skill lists the modules but this
overlay supplies the when.
5. 困境决策案例 (Dilemma cases)
Case A — Arithmetic in prose hallucinates → PoT fixes it (under-coding)
Trigger: A finance-summary agent step: "Given these 14 line items, compute the total,
the 8.25% tax, and the grand total." The agent is a dspy.ChainOfThought module emitting prose.
Constraints:
14 multi-digit addends + a percentage + a sum-of-sums. Well outside reliable mental-math range.
The number flows into an invoice — wrong is expensive and silent.
A Python sandbox is available.
Decision steps:
Gate (§3 Step 1): "Is there a single verifiable answer?" Yes — the total is a fact, not a judgment. → computable.
Recognize the under-coding failure mode. Prose chain-of-thought will emit a plausible
running sum that is frequently off by some digits, and no exception will fire. This is the
silent failure the gate exists to prevent.
Switch the module from ChainOfThought to ProgramOfThought. The model now emits
subtotal = sum([...]); tax = round(subtotal * 0.0825, 2); total = subtotal + tax and the
interpreter computes it exactly.
Feed the result back (op #4): the LM receives total = 4,217.93 and narrates the invoice
line. Code computed; LM contextualized.
Serialize per [[agentsop-output-format-by-model]]: the generated code goes in a fenced block, not a
JSON program field — code-in-JSON would degrade it.
Outcome: The arithmetic is now deterministic and auditable. PoT-style code execution is the
documented fix for exactly this class of error [arXiv 2211.12588, arXiv 2211.10435].
Extractable operation: Multi-step arithmetic in prose is a smell. Re-route it to code (op #6).
Case B — Over-coding a judgment task wastes a sandbox round-trip (over-coding)
Trigger: A support-triage agent step: "Read this customer message and decide whether the
tone is hostile, neutral, or warm." An over-eager engineer wires it through ProgramOfThought
because "code is more reliable."
Constraints:
"Tone" has no single verifiable answer — it is a judgment.
The sandbox round-trip adds latency + cost.
Forcing it into code produces something like if "!!!" in msg: tone = "hostile" — a brittle
rule that encodes a subjective rubric as if it were a formula, and is worse than the model's
native judgment.
Decision steps:
Gate (§3 Step 1): "Single verifiable answer a program could check?" No — tone is
judgment. → reason in prose. Stop.
Invoke the over-coding veto (op #7). There is no deterministic core to delegate to an
oracle. Code adds no determinism — it just relocates the same judgment into worse,
hard-coded heuristics.
Cost it out. The sandbox call adds a full round-trip (and a code-gen + parse step) for
zero precision gain. This is pure waste — the loud, cheap failure mode.
Keep it as ChainOfThought (or Predict). The LM is the right engine for judgment.
Counter-check: if a later spec adds "and count how many exclamation marks" — that
sub-step is computable; decompose (op #2) and route the count to code, the tone to prose.
Outcome: No sandbox call. The judgment stays where judgment belongs. Over-coding is a real
and common anti-pattern: not everything benefits from an interpreter, only things with a
verifiable deterministic core.
Extractable operation: No verifiable answer → no code. Veto the sandbox round-trip (op #7).
Case C — Mixed step: summary + total (decompose)
Trigger: "Summarize this quarter's sales narrative and give me the exact total revenue."
Constraints: One sentence contains both a judgment (summary) and a computation (total).
Decision steps:
Gate → mixed. Decompose (op #2).
Route the total to ProgramOfThought: code sums the figures, interpreter verifies.
Route the summary to ChainOfThought: prose synthesis, no oracle exists.
Feed the code result back into the prose step so the narrative cites the exact, verified total.
Outcome: Each sub-step uses its correct engine. This is the execution-decision analogue of
the mixed-content two-pass pattern in [[agentsop-output-format-by-model]] §5 Case B.
Extractable operation: One task ≠ one strategy. Gate per step, decompose mixed steps.
6. 反模式与边界 (Anti-patterns & boundaries)
Anti-patterns
Code for everything ("code is always more reliable"). False. Code is more reliable only
where a deterministic, verifiable answer exists. For judgment tasks, code just hard-codes a
subjective rubric and adds a wasted round-trip (Case B). The interpreter is an oracle only for
computable questions.
Reasoning for arithmetic ("the model can just add it"). The headline failure. LMs are
unreliable calculators; multi-step prose arithmetic hallucinates plausible wrong numbers,
silently (Case A). Any non-trivial computation → code [arXiv 2211.12588].
Emitting code but never feeding the result back. PoT is "code computes, LM contextualizes."
If the value never re-enters the LM turn, you have a dangling computation the agent can't use
or narrate (op #4).
Unbounded error-retry loops. On a traceback, regenerate-and-rerun is correct — but cap it
(max_iters ≈ 3) and fall back to prose or surface the failure. Infinite code-repair loops
burn cost. See [[agentsop-test-fix-loop]].
Nesting generated code inside JSON sub-structure. The execution decision says emit code;
[[agentsop-output-format-by-model]] says emit it as a fenced block / single string, never as nested
JSON — code-in-JSON degrades the code (Aider 61%→20%). Pass both gates.
Treating LangChain PythonREPLTool as a safe sandbox. It runs in-process, unsandboxed.
Fine for trusted self-authored code; hostile to untrusted input. Use a real sandbox
(OpenAI Code Interpreter container, Anthropic code-exec VM, DSPy's interpreter) when inputs are untrusted.
Over-decomposing trivial computation. A 3-item count or single-digit add does not need a
sandbox round-trip; the tax exceeds the benefit. The gate is for non-trivial computation.
Boundaries (when this gate doesn't fire)
No sandbox available and determinism is soft. Sometimes you must reason in prose because
there is no interpreter; accept the precision risk and flag it.
The model is overwhelmingly capable for a tiny computation. Frontier models reliably handle
small arithmetic; the round-trip isn't worth it below a complexity threshold.
The task is one-shot exploration with no downstream consumer of the precise value.
Latency-critical paths where a sandbox round-trip violates a hard SLA and an approximate
answer is acceptable — a deliberate, documented tradeoff, not a default.
7. 跨框架对照 (Ecosystem cross-reference)
How major frameworks expose the emit-code-vs-reason mechanism. This skill operates at the
decision layer; each framework supplies the mechanism.
The canonical declarative version. A signature compiled with dspy.ProgramOfThought(Sig)
makes the LM emit Python, runs it in an interpreter sandbox, and feeds the result back —
with bounded retry on error (max_iters). The sibling dspy skill lists ChainOfThought
vs ProgramOfThought as module choices but does not give the decision rubric; this overlay
is that rubric. Use ProgramOfThought exactly when §3 Step 1 returns "computable."
OpenAI Code Interpreter / Assistants code tool
A managed container that the model can write Python into and execute, with files and state
persisted across turns. Heavier and stateful — good for data-analysis sessions (load CSV,
compute, plot). Same gate applies: route computable steps in, keep judgment in chat.
Anthropic code-execution tool
A sandboxed VM exposed as a tool; the model emits code, it runs, results return to the
conversation. First-party, sandboxed — safe for untrusted inputs. The execution-decision gate
maps directly: offer the tool, but the model/agent should only invoke it for computable steps.
LangChain PythonREPLTool
A tool wrapping a Python REPL. Runs in-process and is unsandboxed — powerful and dangerous.
Use only for trusted, self-authored computation; never expose it to untrusted input without an
external sandbox. The decision rubric is identical; the safety profile is worst-in-class.
Every framework can be mis-invoked — pointed at a judgment task (over-code) or skipped for a
computation (under-code). This overlay is the gate that decides invocation, regardless of which
mechanism is underneath.
Quick reference card
┌──────────────────────────────────────────────────────────────────────┐
│ EMIT-CODE-VS-REASON DECISION CARD │
├──────────────────────────────────────────────────────────────────────┤
│ Single verifiable answer a program could check? │
│ YES → EMIT CODE → sandbox → run → feed result back into LM │
│ NO → REASON IN PROSE (no oracle exists for judgment) │
│ MIXED → decompose; route each sub-step independently │
├──────────────────────────────────────────────────────────────────────┤
│ Arithmetic / parse / sort / count / regex / symbolic → CODE │
│ Tone / quality / summary / design / synthesis → PROSE │
│ "summary AND total" → SPLIT │
├──────────────────────────────────────────────────────────────────────┤
│ LMs are unreliable calculators but reliable coders. │
│ Under-coding fails SILENTLY (wrong plausible number). │
│ Over-coding fails LOUDLY+CHEAPLY (wasted sandbox round-trip). │
│ When genuinely uncertain AND a verifiable core exists → lean CODE. │
├──────────────────────────────────────────────────────────────────────┤
│ NEVER: │
│ • code a judgment task (over-coding veto) │
│ • do multi-step arithmetic in prose (under-coding) │
│ • forget to feed the code result back into the LM │
│ • nest generated code in JSON (see [[agentsop-output-format-by-model]]) │
│ • loop code-repair unbounded (see [[agentsop-test-fix-loop]]) │
│ • trust LangChain PythonREPLTool on untrusted input (unsandboxed) │
└──────────────────────────────────────────────────────────────────────┘
引用源 (Citations)
Primary anchors:
Program of Thoughts (PoT) — Chen et al., arXiv 2211.12588 — decouple computation from reasoning; LM writes code, interpreter computes.
PAL: Program-aided Language Models — Gao et al., arXiv 2211.10435 — LM as coder, runtime as deterministic solver.
dspy-sop-skill/SKILL.md — ships ProgramOfThought but not this decision rubric (the gap this overlay fills).
d-output-format-by-model-skill/SKILL.md — sibling: once you emit code, how to serialize it (PoT for math/parse; code never nested in JSON). Cross-linked as [[agentsop-output-format-by-model]].
test-fix-loop — the execute → error → retry loop this skill defers to for bounded code repair. Cross-linked as [[agentsop-test-fix-loop]].
