| name | agent-degraded-output-suspect-execution-environment |
| description | When a capable agent first declares a STRONG plan/contract/approach then a retry or continuation produces a WEAKER one, suspect the execution environment couldn't run the strong version — not model laziness or a prompting gap. |
Agent declares strong, then degrades → suspect the execution environment
Problem
An LLM agent with a declare→execute split (declares a verification
contract / a plan / a tool sequence, then carries it out) produces a
weak final artifact. The obvious first hypothesis — "the model is
lazy / the prompt is too soft / it doesn't follow instructions" — is
often wrong.
- 증상: the agent declares something strong (
pytest, a real test
command, a thorough plan), the run fails or hits a round cap, and a
retry / continuation re-declares a weaker version (py_compile,
an import-only check, a trivial plan) that does pass.
- 근본 원인: the strong version was un-runnable in the agent's
execution environment — a missing toolchain, a read-only path, a
permission wall, a placeholder that didn't resolve. The agent ran
the strong command, saw it fail, and rationally degraded to
something the broken environment could satisfy.
- 흔한 오해: blaming prompt strength → you strengthen the prompt, the
agent declares strong AGAIN, it still can't run it, and it degrades
AGAIN. Prompt engineering cannot fix an environmental wall.
Real case (BSNexus, 2026-05): the work LLM's shell_exec ran in a
backend container with no pytest/ruff. qwen3 declared a strong
pytest contract (A1 prompt work succeeded), spent 36 rounds unable
to run it (pip install blocked — read-only site-packages), hit the
loop cap; the Tier-1 continuation then re-declared an import-only
contract that matched the broken environment. The deliverable reached
verified with its tests never run. The fix was environmental (give
the work phase the toolchain — a sandbox), not prompting.
Solution
When you see declare-strong-then-degrade, diagnose the environment
before touching the prompt:
- Take the STRONG thing the agent declared on the first attempt and
run it yourself, by hand, in the exact environment the agent
runs in (same container, same user, same cwd, same PATH).
- If it fails — missing binary (exit 127),
Permission denied,
read-only FS, an unresolved <placeholder> token — that is the
root cause. The agent isn't lazy; it's adapting to a broken env.
- Fix the environment (install the toolchain, fix ownership, resolve
the placeholder, mount the volume) — not the prompt.
- Re-verify: the agent should now declare strong AND the strong
version should run.
Inspect the agent's tool-call trace (e.g. tool_events): a run that
shows the agent trying the strong command, getting errors, then
pip install attempts, then falling back — is the signature.
Key Insights
- A capable agent degrading its own output is usually rational
adaptation to a broken environment, not a compliance failure. The
agent saw the strong path fail and routed around it.
- The declare→execute split makes this sneaky: the declaration looks
fine (prompt worked), so you don't suspect the prompt is innocent.
The break is between declare and execute.
- Prompt engineering has a hard ceiling here — re-nudging a model to
declare strong does nothing when the environment can't run strong.
Same family as
local-llm-runtime-nudge-ceiling.
- First diagnostic move: run the agent's own first-attempt strong
command by hand in its environment. One command tells you env-vs-
prompt.
Red Flags
- A retry / continuation / fallback attempt produces a simpler or
weaker artifact than the first attempt.
- The agent's trace shows it tried tool/command X, got an error, then
switched to a weaker Y.
pip install / npm install / apt attempts mid-run (the agent
is trying to repair a missing toolchain itself).- "The model just won't do X" after you've already strengthened the
prompt once.
- A verification / gate passes via a check that doesn't actually
exercise the deliverable (compile-only, import-only,
--help).
