| name | fastworkflow-debugging-playbook |
| description | Symptom-to-triage playbook for fastWorkflow runtime, build, and train failures. Load this when you see: the wrong command being matched, "threshold.json not found" / FileNotFoundError in ___command_info, the internal CME workflow retraining on every `fastworkflow train`, parameter extraction looping on NOT_FOUND, the CLI hanging or never printing an agent question, FastAPI 409/202 surprises or "poll /turns" advice, DeprecationWarning about process_message, commands with too few training utterances, stale/frozen LLM outputs (DSPy cache), RocksDB/speedict lock errors, env vars that "don't take effect", or tests that mysteriously pass/fail/skip. Do NOT load this for writing new tests (fastworkflow-validation-and-qa), env-var reference (fastworkflow-config-and-flags), historical post-mortems (fastworkflow-failure-archaeology), or tau2 benchmark work (tau2-reliability-campaign).
|
TEAM-PRIVATE: embeds content from uncommitted internal docs. Do not commit or publish this skill without the developer'sexplicit approval.
fastWorkflow Debugging Playbook
Runbook for diagnosing fastWorkflow failures. Written against v2.22.2 (commit c33b9a5).
Every file:line below was verified against that tree; re-verification one-liners are in
Provenance and maintenance.
When to use / when NOT to use
Use when something is broken and you need symptom → cause → discriminating experiment → fix.
Do NOT use — go to a sibling instead:
| Need | Sibling skill |
|---|
| Full incident chronicles (root cause + evidence + status) | fastworkflow-failure-archaeology |
| What every env var / CLI flag does | fastworkflow-config-and-flags |
| How to run CLI/examples/FastAPI server normally | fastworkflow-run-and-operate |
| Measurement scripts and interpretation guides | fastworkflow-diagnostics-and-tooling |
| Intent/param-extraction internals (models, thresholds, DSPy) | fastworkflow-nlu-pipeline-reference |
| Adding tests, what counts as evidence | fastworkflow-validation-and-qa |
| Change gating / non-negotiables before you fix anything | fastworkflow-change-control |
| tau2/tau-bench experiment work (E0–E25) | tau2-reliability-campaign |
Jargon (defined once)
- Workflow: a directory with
_commands/*.py command files; fastWorkflow wraps your app with it.
- CME: the internal
command_metadata_extraction workflow (in fastworkflow/_workflows/) that runs intent detection + parameter extraction as a pseudo-command named wildcard. Every session owns a private CME workflow instance.
- WEC:
WorkflowExecutionContext (fastworkflow/workflow_execution_context.py) — the transport-free execution core. One user message in, one turn out.
- Topology A / B: A = CLI
ChatSession with queues + a worker thread (blocking ask_user); B = bare WEC embedded by FastAPI (ask_user suspends and resumes on the next message).
___command_info/: per-workflow trained artifacts — command_directory.json, routing_definition.json, <cmd>_param_labeled.json, and per-context model folders (tinymodel.pth/, largemodel.pth/, label_encoder.pkl, threshold.json).___convo_info/: per-workflow runtime NLU caches (learned utterance→command mappings, suggested commands) in RocksDB (speedict.Rdict) files.- Fingerprint: sha256 over the set of
(path, size, mtime_ns) of all command sources, stamped into the two JSON artifacts (v2.22.1) so stale snapshots are rebuilt.
- NOT_FOUND: sentinel string (from env var
NOT_FOUND) meaning "parameter not extracted yet"; extraction control-flow branches on it.
- DSPy: LLM-programming library used for parameter extraction and the agent; it caches LLM calls on disk (
~/.dspy_cache for dspy 3.2.1).
Universal first moves
source .venv/bin/activate — always (AGENTS.md rule; wrong interpreter = phantom failures).- Set
LOG_LEVEL=DEBUG in the workflow's fastworkflow.env file, not the shell — get_env_var defaults shadow the OS environment (see trap T11).
- Know which phase you are in: build (AST codegen, no LLM), train (utterance gen + BERT fine-tune →
___command_info/), run (intent → params → execute). A "run" failure is often a train-phase artifact problem.
- Read
fastworkflow-change-control before touching anything under fastworkflow/examples/*/___command_info or running git/bd commands.
Master triage table
| # | Symptom | Most likely cause | Discriminating experiment | Fix / workaround |
|---|
| T1 | Wrong command matched, or import error naming a command file that no longer exists | (a) stale persisted JSON snapshots (pre-v2.22.1 artifacts), (b) in-process lru_cache staleness in a long-running process, (c) a learned bad mapping in ___convo_info, (d) genuine misclassification | Compare compute_commands_source_fingerprint() output vs the source_fingerprint field inside both JSONs; check process uptime vs file edit time; grep the utterance in the ___convo_info cache | (a) auto-fixed on next load in ≥2.22.1; (b) restart the process or call RoutingRegistry.clear_registry(); (c) delete ___convo_info/; (d) retrain / add plain_utterances |
| T2 | Crash or FileNotFoundError on ___command_info/<ctx>/threshold.json; first message fails | Untrained workflow (or a test/experiment wiped the trained model — fix-0hb) | python -c call to is_workflow_trained() (below) | fastworkflow train <wf> <env> <passwords>; for the bundled hello_world: fastworkflow train ./fastworkflow/examples/hello_world ./env/.env ./passwords/.env |
| T3 | fastworkflow train <your-workflow> also retrains fastWorkflow's internal CME workflow, every single time | is_fast_workflow_trained checks ___command_info/ErrorCorrection/largemodel.pth, but training has excluded the ErrorCorrection context since v2.7.0 — the file is never produced, so the check can never pass | ls fastworkflow/_workflows/command_metadata_extraction/___command_info/ → only IntentDetection and global model folders exist | Known mismatch; verified in code 2026-07-09 (see T3 detail). No fix on main. Workaround: none safe; do NOT hand-create the folder. Budget the extra CME train time |
| T4 | Parameter extraction loops: keeps replying "missing information" with fields stuck at NOT_FOUND | Field metadata too weak for the LLM extractor, agent parroting examples values (deliberately rejected), or the sentinel env vars changed | Try the same command with explicit <param>value</param> XML (agent mode parses this by regex before any LLM call); check MISSING_INFORMATION_ERRMSG/NOT_FOUND env values are the template defaults | Improve Field(description=, examples=, pattern=) in the command's Signature.Input; in deterministic mode type abort to escape the state |
| T5 | CLI spinner hangs forever in agent mode when the agent asks the user a question | OPEN, NEEDS RUNTIME VERIFICATION — fix-5fv hypothesis: blocking ask_user enqueues the question on the output queue with no trace sentinel, and the CLI drains the trace queue for a sentinel before reading output | Run any agent-mode CLI session and force a clarification (omit a required param). If it hangs: hypothesis confirmed — update bd fix-5fv | No shipped fix. Workaround: --assistant mode, or /-prefix the command (deterministic path emits its sentinel at turn end) |
| T6 | FastAPI returns 409 on a retry; or 202 with a turn_key and docs say "poll /turns" but GET /turns 404s | 409 = a different turn is already active on the channel (by design). 202 = deferred execution; GET /turns does not exist — fix-85g Step 2 is unbuilt | grep -n "@app.get" fastworkflow/run_fastapi_mcp/__main__.py — routes are /, /probes/*, /conversations only | Recover a 202 by re-sending the byte-identical request — the idempotency key rejoins the same execution. Never treat runtime.lock.locked() as the busy signal (see T6 detail) |
| T7 | DeprecationWarning: WorkflowExecutionContext.process_message() is deprecated in CLI runs | Expected: ChatSession itself still calls process_message internally | grep -n "process_message" fastworkflow/chat_session.py → lines 361, 374, 452, 456 | Harmless; not your bug. New embedder code should call process_turn() |
| T8 | An intent model is weak for one specific command; that command has almost no training utterances | generate_diverse_utterances returns [] on litellm RateLimitError — dropping even the seed utterances for that command, silently (one logger.error line) | grep the train log for LiteLLM Rate limiting error!; inspect utterance counts per command in ___command_info/command_directory.json | Re-run fastworkflow train when not rate-limited; consider a slower model/key. No retry logic exists on main |
| T9 | LLM outputs look frozen — refine/agent/param-extraction returns identical answers despite prompt/code changes | DSPy disk/memory cache serving stale completions | Delete or disable the cache and re-run; if output changes, it was the cache | rm -rf ~/.dspy_cache (dspy 3.2.1 location) or dspy.configure_cache(enable_disk_cache=False, enable_memory_cache=False). WARNING: docs/DSPY_CACHE_GUIDE.md is partly rotten (see T9 detail) |
| T10 | RocksDB errors ("IO error ... LOCK", "Resource temporarily unavailable") or mysterious LOG/.sst files appearing | Two processes/threads opened the same Rdict (RocksDB is single-writer), or you are looking at normal RocksDB internals | fuser/lsof on the .db dir; check for a second server/test process on the same workflow | Ensure one writer per channel/workflow; kill the other process. LOG/.sst/MANIFEST files inside ___convo_info and ___workflow_contexts are normal RocksDB internals, all gitignored — never commit, safe to delete only with the owning process stopped |
| T11 | An env var you exported in the shell is silently ignored (e.g. SESSION_STATE_STORE, INTENT_DETECTION_TINY_MODEL, LOG_LEVEL) | get_env_var(var, type, default) returns the code default without consulting os.environ whenever a default is supplied | Put the var in the workflow's fastworkflow.env and re-run: if it now takes effect, you hit the precedence quirk | Always configure via the env files. Shell exports only work for vars whose call sites pass no default |
| T12 | JWT weirdness: tokens accepted that shouldn't be, or alg=none in decoded tokens | Default server mode is trusted: signatures NOT verified, unsigned tokens use JWT alg="none"; the fastworkflow run_fastapi_mcp CLI cannot enable verification (flag not forwarded) | Decode a minted token; check server launch command | For signed JWTs run python -m fastworkflow.run_fastapi_mcp ... --expect_encrypted_jwt directly. Treat default deployments as trusted-network only |
| T13 | Test suite: tests "pass" on a fresh clone but are mostly skips; or 4 FastAPI tests fail with missing threshold.json; or the suite is ~4 min slower than it should be | Env-gated skips (need ./env/.env + ./passwords/.env + locally pre-trained hello_world); fix-0hb-class model wipe; deliberate 0.5 s sleep after every test | python -m pytest -rs and read skip reasons; check fastworkflow/examples/hello_world/___command_info/global/threshold.json exists | Provision env files, pre-train hello_world (T2 command). Never "fix" the sleep in tests/conftest.py:111-118 without solving ChatWorker thread drain |
Trap details
T1 — Wrong command matched / stale command caches
Intent detection tries, in order (fastworkflow/_workflows/command_metadata_extraction/intent_detection.py:99-133):
exact first-token match → Levenshtein fuzzy match (threshold 0.3) → embedding cache match at 0.85 cosine
(cache_match, fastworkflow/cache_matching.py:131) → transformer CommandRouter.predict. Distinguish four causes:
- Stale persisted snapshots (fixed in v2.22.1, commit b5747df): before the fix,
routing_definition.json
had no staleness check and command_directory.json used newest-mtime (blind to deletions/renames);
add/remove/rename of a command caused import errors until artifacts were hand-deleted.
Now compute_commands_source_fingerprint (fastworkflow/command_directory.py:627) stamps both JSONs and
RoutingRegistry._persisted_definition_is_fresh (fastworkflow/command_routing.py:385) forces rebuild on mismatch.
Experiment: check the source_fingerprint field in both JSONs matches a fresh
compute_commands_source_fingerprint(<wf>) call. A missing field = pre-2.22.1 artifact → rebuilt on next load.
- In-process cache staleness (still open):
get_cached_command_directory is @lru_cache(maxsize=32)
(fastworkflow/command_directory.py:657-658) and RoutingRegistry.get_definition returns
cls._definitions[path] when present (fastworkflow/command_routing.py:357-360) — the fingerprint is only
checked on cache miss. Editing commands while a server/CLI process is running is never picked up.
Experiment: restart the process; if behavior changes, this was it.
Fix: restart, or fastworkflow.RoutingRegistry.clear_registry() (clears definitions, the global command-class
cache, the command-directory lru_cache, and the module import cache — command_routing.py:410-425).
Note the fingerprint embeds absolute paths + mtime_ns, so moving a workflow dir or a fresh checkout triggers a
harmless JSON rebuild (not a retrain).
- Learned clarification cache: a successful "you misunderstood"-style clarification stores the ORIGINAL
utterance → resolved command with its embedding (
intent_detection.py:151-163 calling store_utterance_cache),
so future messages ≥0.85 similar skip the classifier. One bad learned mapping = persistently wrong command.
Experiment/fix: delete the workflow's ___convo_info/ directory (regenerated empty; you lose learned mappings).
- Genuine misclassification: only after 1–3 are excluded. Then it is an NLU-quality problem —
see
fastworkflow-nlu-pipeline-reference (thresholds, wildcard negative class, utterance economics).
T2 — Missing threshold.json / untrained-workflow late failures
threshold.json per context folder is the marker of a trained intent model. Guards exist at two entry points
(fastworkflow/run/__main__.py:156-172 and fastworkflow/mcp_server.py:249-250 — both call
is_workflow_trained, fastworkflow/model_pipeline_training.py:624), added in v2.19.0 (commit d7853a1) after
untrained workflows crashed on the first message instead of at startup. If you still see a late failure, you
came in through a path without the guard (e.g. embedding WEC directly, or a test).
Story: the fix-0hb incident (commit fa97b48) — a training test's teardown rmtree'd the real
fastworkflow/examples/hello_world/___command_info, poisoning the NEXT suite run with exactly this error.
Discipline rule (non-negotiable): never let tests/experiments train into or delete the bundled examples'
___command_info — copytree into a temp dir first (pattern: tests/test_train_modern_stack.py:129-139).
Quick check:
source .venv/bin/activate && python -c "
from fastworkflow.model_pipeline_training import is_workflow_trained
print(is_workflow_trained('<workflow_path>'))"
T3 — CME workflow retrains on every fastworkflow train
Verified state on c33b9a5 (2026-07-09):
is_fast_workflow_trained (fastworkflow/train/__main__.py:225-256) requires
_workflows/command_metadata_extraction/___command_info/ErrorCorrection/largemodel.pth.train() excludes ErrorCorrection from CME training (fastworkflow/model_pipeline_training.py:821,
literally - {'ErrorCorrection', 'ErrorCorrection'} — note the duplicate literal), so that folder is never produced;
the repo's CME ___command_info/ contains only IntentDetection/ and global/.- Therefore the check returns False and
train_main (train/__main__.py:283-287) retrains the internal CME
workflow before your workflow whenever the target path does not contain the substring "fastworkflow"
(the guard is a substring test — it also skips CME training for any user path that happens to contain
"fastworkflow", a separate sharp edge).
This mismatch dates to v2.7.0 and is not fixed on main; whether it is an accepted cost or an unnoticed
regression is an open question for Dhar (also flagged in discovery). Do not "fix" it ad hoc — the correct check
is a design decision (fingerprint vs mtime vs context list); file/attach a bd issue first (one bd write at a
time, verify .beads/issues.jsonl after — bd's close --reason has silently failed to persist before, observed 2026-06-11).
T4 — Parameter extraction NOT_FOUND loops
Mechanics: partial params are stored with NOT_FOUND sentinels via model_construct (no validation)
(fastworkflow/_workflows/command_metadata_extraction/parameter_extraction.py:93-104); validation error messages
embed the MISSING_INFORMATION_ERRMSG / INVALID_INFORMATION_ERRMSG env-var text (fastworkflow/utils/signatures.py:634),
so changing those env values changes user-facing errors AND any code that matches on them.
Landmine: the string-splitting recovery helper _extract_missing_fields (parameter_extraction.py:140-160) is
dead code with an arity bug (3-tuple unpack of validate_parameters' 4-tuple return) — it raises ValueError
if revived; do not call it.
Agent mode first tries XML-regex extraction (_extract_parameters_from_xml, parameter_extraction.py:296+) and
deliberately rejects the result when a value equals one of the field's examples (anti-parroting,
parameter_extraction.py:330-337); rejected → falls back to DSPy LLM extraction, whose BestOfN(N=3) reward
also scores 0.0 for echoed example values (fastworkflow/utils/signatures.py). So an example value that is also a
legitimate input can never be extracted in agent mode — pick example values that are not real data.
Loop-breaker in deterministic mode: type abort (or you misunderstood) — these escape hatches are matched by
exact/fuzzy utterance only, never by the classifier. Agent exhaustion caps at max_iters=25
(fastworkflow/workflow_agent.py:387) → failure_reason="max_iters_exhausted".
T5 — ask_user suspension oddities (incl. the OPEN fix-5fv CLI hang)
Two topologies, two behaviors:
- Topology B (FastAPI/WEC):
ask_user raises AskUserSuspend (fastworkflow/utils/react.py:18) — a
BaseException subclass so the ReAct loop's except Exception cannot swallow it; WEC returns an
awaiting_user output and the next message resumes the same logical turn. A message during suspension is the
answer — it never starts a new turn. CommandCancelledError (workflow_execution_context.py:42) is BaseException
for the same reason. Never convert these to Exception subclasses or add broad except BaseException.
- Topology A (CLI):
_ask_user_tool (fastworkflow/workflow_agent.py:354-385) puts the question on
command_output_queue and blocks on user_message_queue.get().
fix-5fv (P2, OPEN, code-reading evidence only — NEEDS RUNTIME VERIFICATION, never runtime-reproduced):
the blocking path emits no trace sentinel; the only sentinel emitter is _maybe_enqueue_trace_sentinel
(workflow_execution_context.py:656), called only at turn-END sites. The CLI (fastworkflow/run/__main__.py:231-260)
drains the trace queue until a None sentinel BEFORE reading the output queue → predicted deadlock: spinner spins,
question never prints. Filed 2026-06-11 during the R43 design review. If you hit T5, you are the runtime
verification — record the result in bd fix-5fv. (The bead cites pre-refactor line numbers; current ones are above.)
Related invariant: every mid-turn clarification enqueue must be paired with a trace sentinel (design rule A19).
T6 — FastAPI 409 / 202 semantics (turns engine, v2.22.0)
Contract (module docstring, fastworkflow/run_fastapi_mcp/turns.py:1-30 — treat as law):
- wait-or-defer, never wait-or-abort: a request timeout never cancels the execution (it runs as its own asyncio.Task).
- 200 = finished within the wait window; 202 = still running, body carries
turn_key; 409 = a different
turn is active on this channel (ChannelBusyError → 409 at run_fastapi_mcp/__main__.py:607-614).
- Idempotency key = sha256(channel_id + kind + normalized args) (
turns.py:~84-96): a byte-identical retry
rejoins the same execution. That is the ONLY recovery path today: GET /turns does not exist (docstrings
at __main__.py:652 say "poll /initialize or /turns" — Step 2 of fix-85g is unbuilt; the only GET routes are
/, /probes/healthz, /probes/readyz, /conversations). TurnRegistry.evict_terminal is a documented no-op
(turns.py:226-243), so terminal executions accumulate in memory until restart.
- The busy signal is the registry's per-channel active-execution pointer, never
runtime.lock.locked() — the
lock is released while deferring and across AWAITING_USER. Story: the fix-85g 504 race (commit cf3eeae) — a 504
unwound async with runtime.lock, a retry passed the lock-based guard, and two executions raced / tokens returned
with silently-empty startup output. Known survivor of the old pattern: /invoke_agent_stream still checks
runtime.lock.locked() (__main__.py:963) and bypasses the registry — expect inconsistent busy behavior on that
endpoint (open question whether intentional).
- Restart loses all in-flight executions (Step 3 durability unbuilt).
T8 — Silent utterance drops on RateLimitError during train
fastworkflow/train/generate_synthetic.py:120-122: on litellm.exceptions.RateLimitError the function logs one
error line and return [] — the command trains with neither synthetic nor seed utterances (contrast the
no-datasets path at :26-32, which correctly returns [command_name] + seed_utterances). Result: a silently
underrepresented class in the intent model; the train run still "succeeds". Always grep train logs for
LiteLLM Rate limiting error! before trusting a freshly trained model.
T9 — DSPy cache confusion
docs/DSPY_CACHE_GUIDE.md is the doc of record but partly rotten (verified 2026-07-09): it imports
fastworkflow.run_agent.agent_module (module deleted) and invokes dspy_cache_utils.py by bare filename —
the utility itself does exist, at fastworkflow/utils/dspy_cache_utils.py, so the guide's commands fail
as written from repo root. Run it as:
.venv/bin/python -m fastworkflow.utils.dspy_cache_utils status
(matches fastworkflow-diagnostics-and-tooling and fastworkflow-nlu-pipeline-reference). As a CLI only
status and clear-disk are useful — clear/reset call dspy.configure_cache inside a fresh process
and cannot affect any other running process.
What still works from the guide:
import dspy
dspy.configure_cache(enable_disk_cache=False, enable_memory_cache=False)
rm -rf ~/.dspy_cache
Symptom signature: you changed a prompt/signature/field description but outputs are byte-identical across runs.
Also remember tests and the flaky "LLM-training routing test" (accepted-flaky per commit 033132f message) can be
cache-sensitive.
T10 — speedict/RocksDB notes
Where Rdict still lives (post v2.21.4 hot-path removal, commit 6880b0a): the enablecache decorator
(fastworkflow/workflow.py:57), NLU caches (fastworkflow/cache_matching.py:67,148,
intent_detection.py:200-274), the conversation store (run_fastapi_mcp/conversation_store.py:45).
All open/close per operation because RocksDB is single-writer — two concurrent writers on one DB error out
(hence the "one writer per channel / sticky routing" rule; enforcement is open issue fix-5ka).
LOG, *.sst, MANIFEST-*, CURRENT, LOCK files inside ___convo_info/, ___workflow_contexts/ are normal
RocksDB internals (you will find such droppings under tests/todo_list_workflow/___workflow_contexts/ — on-disk
only, gitignored, untracked). Do not commit them; delete only when no process has the DB open. A leftover LOCK
after a crash: confirm no live process (lsof +D <dbdir>) before removing the directory.
T11 — env sharp edges
get_env_var (fastworkflow/__init__.py:211-219): checks the init()-loaded dict first; if absent and a default
was supplied, returns the default without consulting os.environ. Consequences:
- Vars with code defaults (
SESSION_STATE_STORE, INTENT_DETECTION_TINY_MODEL, INTENT_DETECTION_LARGE_MODEL)
can only be overridden via the env files.
build/refine call fastworkflow.init(env_vars={}) — they read no env files; their LLM vars
(LLM_COMMAND_METADATA_GEN, LITELLM_API_KEY_COMMANDMETADATA_GEN — note the no-underscore spelling) must be OS
exports, unlike train/run.- Never read env vars at import time in new code: the v2.21.3 regression (commit 79e6986) crashed the server when
SPEEDDICT_FOLDERNAME lived only in the env file, and was masked in dev by litellm's load_dotenv of the cwd .env.
run hard-fails if LITELLM_API_KEY_SYNDATA_GEN is missing (run/__main__.py:131-132) — it is used as a
file-presence probe, so Bedrock/proxy users hit it even though run does no synthetic generation.
T13 — test-suite gotchas (debugging context only; authoring → fastworkflow-validation-and-qa)
- No pytest config anywhere; run
python -m pytest from repo root in the venv. ~495 collected.
- Key-gated skips: FastAPI/topology-B/probes/MCP tests skip unless
./env/.env AND ./passwords/.env exist
(tests/test_fastapi_service.py:26-36 pattern) — a green run on a fresh clone is mostly skips.
- 4 FastAPI tests additionally require a locally pre-trained hello_world example (T2 command). fix-0hb
(commit fa97b48) is the incident; the temp-copy training pattern is the law.
tests/conftest.py:111-118 sleeps 0.5 s after EVERY test (ChatWorker thread drain) — ~4 min fixed overhead; a
"hung" suite is usually just this.tests/test_streaming_endpoint.py silently passes with no server running (bare return on connection error) —
zero signal; ignore its "pass".- Bundled
fastworkflow/examples/fastworkflow.passwords.env contains a dead placeholder key that litellm rejects
with AuthenticationError; only repo-local ./passwords/.env has real keys. Placeholder heuristic: values
containing < or your- are not keys.
Cache-layer map (what to clear for which symptom)
| Layer | Scope | Clear with | Symptom it explains |
|---|
RoutingRegistry._definitions + _GLOBAL_COMMAND_CLASS_CACHE + get_cached_command_directory lru | process | RoutingRegistry.clear_registry() or restart | edits to _commands/ invisible in a live process (T1b). Note: command-class cache is keyed command_name:module_type with NO workflow path — command names must be unique per process |
command_directory.json / routing_definition.json | disk, fingerprint-guarded | delete files (auto-rebuilt) — safe | stale routing pre-2.22.1 (T1a) |
___convo_info/ Rdict caches | disk, per workflow | delete directory | learned wrong intent mapping (T1c), stale suggested-commands |
___command_info/ model folders | disk, per workflow | re-run fastworkflow train (never hand-delete parts; deleting the whole folder forces full retrain) | wrong-era models; NOTE: failed retrain deliberately keeps prior artifacts runnable (prune happens only after success) |
| DSPy disk/memory cache | user-global | rm -rf ~/.dspy_cache / dspy.configure_cache(...) | frozen LLM outputs (T9) |
_WORKFLOW_REGISTRY weakref session registry | process | drop references / restart; holding a reference IS the session | "session survived longer/shorter than expected"; CLI has NO cross-restart resume since v2.21.4 |
Discipline rules that bind you while debugging
- Never
git commit/push without the developer'sexplicit request in that turn — rule established 2026-07-08 after a
private doc was auto-pushed to this PUBLIC repo, forcing a history rewrite. No "session-close protocol" overrides it.
- One bd write at a time; verify
.beads/issues.jsonl changed after each; don't trust bd close --reason
(observed silently failing to persist, 2026-06-11).
- Never wipe/train-into
fastworkflow/examples/*/___command_info (fix-0hb, commit fa97b48) — temp copies only.
- tau-bench parity is sacred: never modify tau-bench tools/tasks for benchmark runs; disclose any nonstandard
trade (e.g. pinned simulator temperature) — never silent. (Benchmark debugging →
tau2-reliability-campaign.)
- Doc-rot precedent: trust code over docs. Known rot: CLAUDE.md says intent models are "via scikit-learn"
(actually torch/transformers fine-tuning; sklearn is only LabelEncoder/split/metrics) and omits
tests/todo_list_workflow; docs/DSPY_CACHE_GUIDE.md cites the deleted run_agent.agent_module and
mislocates dspy_cache_utils.py (really at fastworkflow/utils/); run_fastapi_mcp/README.md
documents the pre-turns 504 behavior.
Provenance and maintenance
Facts verified 2026-07-09 against v2.22.2 (commit c33b9a5). fix-5fv and the T3 CME-retrain finding were
re-verified in source on that date; both remain code-level findings without a runtime reproduction (T5) or an
owner ruling (T3). Re-verify volatile facts before relying on them:
grep -n "def compute_commands_source_fingerprint\|@lru_cache" fastworkflow/command_directory.py
grep -n "_persisted_definition_is_fresh\|def clear_registry" fastworkflow/command_routing.py
grep -n "is_workflow_trained" fastworkflow/run/__main__.py fastworkflow/model_pipeline_training.py
grep -n "ErrorCorrection" fastworkflow/train/__main__.py fastworkflow/model_pipeline_training.py
ls fastworkflow/_workflows/command_metadata_extraction/___command_info/
grep -n "MISSING_INFORMATION_ERRMSG\|examples and extracted_value" fastworkflow/_workflows/command_metadata_extraction/parameter_extraction.py
bd show fix-5fv | head -5
grep -n "_maybe_enqueue_trace_sentinel\|def _ask_user_tool" fastworkflow/workflow_execution_context.py fastworkflow/workflow_agent.py
grep -n "@app.get\|lock.locked()" fastworkflow/run_fastapi_mcp/__main__.py
grep -n "self._core.process_message" fastworkflow/chat_session.py
grep -n -A2 "RateLimitError" fastworkflow/train/generate_synthetic.py
python -c "from dspy.clients import DISK_CACHE_DIR; print(DISK_CACHE_DIR)"
ls fastworkflow/utils/dspy_cache_utils.py
sed -n '211,220p' fastworkflow/__init__.py
grep -n "expect_encrypted_jwt" fastworkflow/cli.py || echo "still not forwarded"
sed -n '111,118p' tests/conftest.py
bd show fix-5fv fix-85g fix-5ka 2>/dev/null | grep -E "OPEN|CLOSED"
When any command above stops matching, fix this file in the same change — a wrong runbook is worse than none.
