fastworkflow-diagnostics-and-tooling

name

fastworkflow-diagnostics-and-tooling

description

Load this skill when you need to MEASURE a fastWorkflow workflow instead of eyeballing it — inspect ___command_info trained artifacts, check whether cached command snapshots are stale (fingerprint FRESH/STALE), capture or read a turn's command traces / action.jsonl, smoke-test intent-classifier accuracy, configure LOG_LEVEL, inspect the DSPy LLM cache, or hit the server probe endpoints. Trigger phrases/symptoms: "why did it rebuild", "is this workflow trained", "what did the agent actually do", "traces are empty", "stale cache", "threshold.json", "confusion between commands", "cache hit hides the LLM call". Do NOT load for diagnosing a specific failure end-to-end (use fastworkflow-debugging-playbook), for NLU/model theory (use fastworkflow-nlu-pipeline-reference), or for statistics like pass^k variance (use fastworkflow-proof-and-analysis-toolkit).

fastWorkflow diagnostics and tooling — measure, don't eyeball

Runbook for observing what a fastWorkflow workflow actually did and what state its artifacts are actually in. Ships four tested, read-only scripts in scripts/ plus a map of every observability surface the framework already has. All paths are repo-relative to the fastWorkflow repo root; all facts verified against v2.22.2 (commit c33b9a5).

When to use / when NOT to use

Use this skill when you need to:

• dump/inventory a workflow's ___command_info/ (commands, utterance counts, thresholds, model files, DSPy few-shot artifacts)
• decide "rebuild vs stale vs fine" for command_directory.json / routing_definition.json (the v2.22.1 fingerprint)
• capture or read a turn's command trace: live CLI traces, action.jsonl, the server response's traces field, TurnOutput
• get numbers on a trained intent classifier (per-command accuracy on seeds, confusion pairs, ambiguity/fallback rates)
• control logging verbosity, inspect the DSPy cache, or use the K8s probes

Do NOT use this skill for — go to the sibling instead:

Need	Sibling skill
symptom → root-cause triage of a failure	fastworkflow-debugging-playbook
how the BERT two-tier/threshold/DSPy pipeline works	fastworkflow-nlu-pipeline-reference
env var / CLI flag catalog and defaults	fastworkflow-config-and-flags
starting the CLI / FastAPI+MCP server, artifact locations	fastworkflow-run-and-operate
what counts as evidence, adding tests	fastworkflow-validation-and-qa
pass^k / variance / attribution math	fastworkflow-proof-and-analysis-toolkit
running tau2 experiments E0–E25	tau2-reliability-campaign
whether a change is allowed at all	fastworkflow-change-control

Ground rules (non-negotiable)

• Everything here is read-only. The shipped scripts never write. When a diagnosis leads you to retrain, train into a temp copy of the workflow — never let anything (tests, experiments, teardown) touch fastworkflow/examples/*/___command_info. A training test once destroyed the pre-trained example models other tests silently depended on (the fix-0hb incident, fixed in commit fa97b48).
• Never git commit/push diagnostics output or anything else without Dhar's explicit request in that turn (team rule since 2026-07-08).
• Use the repo venv interpreter: .venv/bin/python. import fastworkflow takes ~5–10 s (pulls dspy/torch); scripts note where that cost applies.

The four scripts

All live in .claude/skills/fastworkflow-diagnostics-and-tooling/scripts/. Each supports --json for machine-readable output (except trace_turn, whose inputs are already JSON). Full real observed outputs with healthy-vs-sick walkthroughs: references/observed-outputs.md.

Script	Question it answers	Cost
inspect_command_info.py <wf>	what is in ___command_info/: commands, utterance counts, parameterized?, *_param_labeled.json valid/rejected counts, per-context model files present/missing, trained threshold values, fingerprint FRESH/STALE	instant with --no-fingerprint, else +~7 s import
check_cache_freshness.py <wf>	will the JSON snapshots be trusted or rebuilt, and why — recomputes the v2.22.1 fingerprint, prints verdict + the 5 most recently modified command sources (the usual culprit)	~7 s (imports fastworkflow)
trace_turn.py {actions,response,explain}	read action.jsonl / a saved server turn-response body; explain prints the capture recipes per topology	instant (stdlib only)
collect_model_metrics.py <wf> [--context C]	per-command top-1 accuracy, confusion pairs, ambiguity rate, DistilBERT-fallback rate of a TRAINED context, scored on stored seed utterances with the exact runtime decision rule	~10–60 s/context (loads 2 transformer models, CPU ok)

Copy-paste smoke test (works offline, no API keys, uses the bundled internal workflow which is trained on any machine that has ever run fastworkflow train):

cd /path/to/fastworkflow
S=.claude/skills/fastworkflow-diagnostics-and-tooling/scripts
.venv/bin/python $S/inspect_command_info.py  fastworkflow/_workflows/command_metadata_extraction
.venv/bin/python $S/check_cache_freshness.py fastworkflow/_workflows/command_metadata_extraction
.venv/bin/python $S/trace_turn.py explain
.venv/bin/python $S/collect_model_metrics.py fastworkflow/_workflows/command_metadata_extraction

Interpretation: inspect_command_info.py

Output pattern	Meaning	Next action
UNTRAINED contexts: ... global	built but never trained here (trained models are never shipped in the wheel and ___command_info is not committed for examples)	fastworkflow train (temp copy if bundled example)
IntentDetection: NO MODEL FOLDER on an app workflow	normal — internal CME contexts are trained inside fastworkflow's bundled command_metadata_extraction workflow, not per app	nothing
ErrorCorrection: NO MODEL FOLDER	normal — ErrorCorrection commands are matched only by exact/fuzzy utterance match, never the classifier	nothing
param_labeled ... MISSING on a params=yes command	DSPy few-shot examples absent → parameter extraction runs zero-shot, silently worse	retrain; check LITELLM_API_KEY_SYNDATA_GEN
MISSING: largemodel.pth (or any model file) in a context marked complete elsewhere	partial artifacts — interrupted train or manual deletion; CommandRouter will crash loading this context	retrain that workflow
thresholds look weird (large_amb=0.0)	legitimate on tiny label sets: ambiguous threshold = mean confidence of that model's misclassifications; zero misclassifications → 0.0	nothing, unless paired with bad metrics
plain: 0 across app commands	command files declare empty plain_utterances	add seeds / run fastworkflow refine

Interpretation: check_cache_freshness.py

Verdict	Meaning	Next action
FRESH	snapshots trusted as-is on next load	nothing
STALE	some command source changed/moved/re-stat'ed since stamping; next load rebuilds JSONs (cheap, automatic)	check the newest-sources list; if commands actually changed since last train, the classifiers have label drift → retrain
STALE but sources byte-identical	fingerprint embeds absolute paths + mtime_ns — fresh checkout, moved dir, Docker COPY, touch	spurious rebuild, harmless; expected on every fresh clone
LEGACY	pre-v2.22.1 snapshot without a fingerprint	will rebuild; fine
MISSING	never built here / info dir deleted	fastworkflow train

Key asymmetry to keep straight: the fingerprint guards only the two JSON snapshots. The trained models (*.pth, thresholds) are not fingerprint-guarded and only change on an explicit fastworkflow train (which prunes orphaned artifacts only AFTER success, so a failed retrain leaves the previous models runnable).

Interpretation: trace_turn.py (and turn records generally)

Trace vocabulary, defined once:

• CommandTraceEvent — in-memory event emitted around every agent tool call (workflow_agent.py:130-139, 209-218): AGENT_TO_WORKFLOW (raw command text the agent sent) then WORKFLOW_TO_AGENT (resolved command_name, parameters, response_text, success). A None sentinel terminates each turn's stream.
• action.jsonl — CLI-only debug mirror in the CWD of fastworkflow run (ChatSession constructs its context with mirror_action_log_to_file=True, chat_session.py:118). Deleted at every turn start (workflow_execution_context.py:711-713): it always holds only the LAST turn. Two record shapes: tool call {command, command_name, parameters, response} and ask_user {agent_query, user_response}.
• TurnOutput — returned by ctx.process_turn(message); carries turn_key (the developer observability handle for one logical turn, stable across ask_user suspensions), status, failure_reason, answer, command_outputs. success = all commands succeeded and is deliberately orthogonal to status (turn.py:196-216).
• Server traces field — the FastAPI turns engine drains the trace queue destructively into the turn response (run_fastapi_mcp/turns.py:293-296); key only present when non-empty.

Pattern	Meaning	Next action
success: false + status: completed + confident answer	a command failed and the agent masked it in prose	find the failing command_outputs entry; fastworkflow-debugging-playbook
failure_reason: max_iters_exhausted	agent hit max_iters=25	read traces for the loop
empty traces on a retry	destructive drain already consumed them (fix-85g Step 2 open)	treat traces as read-once; persist the first response
ask_user entry with success: false	question still unanswered, NOT an error (A7 role inversion: parameters=agent's question, response=user's answer, duration_ms=user think time)	resume the turn
action.jsonl missing	not a CLI run, assistant (non-agent) turn, or already overwritten	use server traces / in-process ctx.action_log instead

Interpretation: collect_model_metrics.py

Scores are on SEED utterances — training data, so near-perfect is the expected baseline. Observed healthy reference (CME workflow): global context top1=1.0; IntentDetection top1=0.9677 with one real confusion (go_up -> reset_context).

Pattern	Meaning	Next action
top1 ≥ ~0.95, few confusions	healthy	nothing
top1 < ~0.9 on seeds	broken/mismatched artifacts or label drift	check_cache_freshness.py; retrain
recurring confusion pair	genuinely overlapping seed phrasings; at runtime becomes the ambiguity-clarification flow	differentiate plain_utterances, retrain
high ambiguous_rate	thresholds above typical confidence → frequent clarification prompts	compare thresholds vs mean_confidence
everything → wildcard	out-of-scope class swallowed commands; artifacts likely from a different command set	retrain
sklearn InconsistentVersionWarning on LabelEncoder	version skew between training sklearn and current venv (observed: 1.7.0 pickle in 1.9.0 — benign here)	if labels look shuffled, retrain

Honest gap: true held-out F1 is not cheaply computable. Synthetic training utterances (litellm + PersonaHub) are generated at train time and never persisted; training's own test-split metrics (score = f1 × ndcg × (1 − 0.15 × distil_usage%/100), model_pipeline_training.py:222-258) exist only in train-time stdout. If you need them, capture train logs. No target quality bars are documented anywhere — treat acceptable-F1 thresholds as an open question (candidate work item for the tau2 program).

The observability map (what already exists, where)

Logging

• Logger name fastWorkflow, configured at import in fastworkflow/utils/logging.py (nanosecond timestamps).
• Set LOG_LEVEL (DEBUG/INFO/WARNING/ERROR/CRITICAL) in the workflow's fastworkflow.env — fastworkflow.init() reconfigures the logger from the dotenv dict (__init__.py:180-182), fixing the old trap where import-time config ignored env-file values. run_fastapi_mcp additionally pre-reads the env file to set uvicorn's log level (run_fastapi_mcp/__main__.py:1611-1617).
• Noise policy: DSPy/LiteLLM/httpx/etc. loggers are force-quieted at utils/logging.py:104-114 (LiteLLM to CRITICAL). If you need LLM-call detail, raise those explicitly in your harness.
• FW_EAGER_ARTIFACT_VALIDATION=0 (OS env, read at turn.py:155) silences the v3.0 artifact-serializability deprecation warnings.

Live CLI traces (agent mode)

fastworkflow run renders CommandTraceEvents while the spinner runs — dim yellow Agent > (raw command), dim yellow/green Workflow > (resolved command + response), dim orange for failed steps. Methodology and event model: docs/agent_mode_live_command_traces.md — the design doc for this feature; its event model and rendering rules match v2.22.2, but note the drift: the proposed --no_agent_traces flag and FASTWORKFLOW_SHOW_AGENT_TRACES override were never implemented (traces are always on; verified by grep), and action.jsonl records are appended by workflow_agent.py, not command_executor.py as §5 claims. Consumer loop: fastworkflow/run/__main__.py:229-267. Known open bug fix-5fv (code-reading evidence only, never reproduced): the CLI spinner may hang on agent-mode ask_user because the clarification is enqueued without the trace sentinel.

In-process capture (the pattern for experiment harnesses)

import queue, fastworkflow
fastworkflow.init(env_vars={...})                     # required before framework classes
ctx = fastworkflow.WorkflowExecutionContext(run_as_agent=True)
trace_q = queue.Queue()
ctx.set_transport_queues(command_trace_queue=trace_q) # queues are injected, not ctor args
# ... start workflow ...
out = ctx.process_turn("user message")                # TurnOutput
out.turn_key, out.status, out.success, out.command_outputs
ctx.action_log                                        # this turn's records
# drain trace_q until the None sentinel for live events

Duck-typing contract: anything passed to CommandExecutor/workflow_agent as chat_session_obj must expose get_active_workflow / cme_workflow / command_trace_queue / append_action_log / append_turn_output / append_ask_user_entry / complete_ask_user_entry; ChatSession and WorkflowExecutionContext both do. Missing members are gracefully no-op'ed — i.e. turn capture is silently dropped, so a custom embedder that "works" but shows empty traces is missing this surface.

FastAPI server

• Turn endpoints (/invoke_agent, /invoke_assistant, /initialize, /perform_action) return turn_key, exec_state, status, success, answer, command_responses, command_outputs, and traces (when non-empty) — shapes in run_fastapi_mcp/turns.py:render_turn_response. 202 {turn_key, exec_state:"running"} = deferred; re-POST the identical body to rejoin; different body → 409.
• /invoke_agent_stream streams trace events as NDJSON (or SSE) but is NOT integrated with the turns registry (no idempotent rejoin) — inconsistent by known omission.
• Probes: GET /probes/healthz → {"status": "alive"}; GET /probes/readyz → {"status": "ready"|"not_ready", "checks": {...}} (fastworkflow initialized + workflow path valid). Both are access-log-suppressed when 200 (__main__.py:93-122), so absence from logs is normal.

DSPy cache

Default: disk+memory cache at ~/.dspy_cache (observed), LiteLLM cache off. Inspect: .venv/bin/python -m fastworkflow.utils.dspy_cache_utils status. Cache hits skip the API and dspy.inspect_history() — clear before debugging LLM behavior. Details, real output, and the doc-rot warning about docs/DSPY_CACHE_GUIDE.md (references a nonexistent fastworkflow.run_agent module): references/observed-outputs.md §5.

Known observability gaps (open at v2.22.2 — do not design around them silently)

Gap	Status
traces are read-once (destructive drain); no replay buffer	fix-85g Step 2, open
no GET /turns/{turn_key} polling endpoint despite docstring references	fix-85g Step 2, open
TurnResult persistence/metrics layer (ConversationTurnStore, MetricsSink, admin CLI)	designed (docs/turn_result_design_final.md), unbuilt
train-time model metrics not persisted anywhere	gap; capture stdout
CLI ask_user spinner hang	fix-5fv, open, unverified

Provenance and maintenance

Facts verified 2026-07-09 against v2.22.2 (commit c33b9a5), by reading the cited sources and by executing every script in this skill against fastworkflow/_workflows/command_metadata_extraction, fastworkflow/examples/hello_world (read-only), and tests/example_workflow. Observed outputs in references/ are real except the one labeled SYNTHETIC.

Re-verification one-liners for volatile facts:

Fact	Re-verify with
fingerprint fields/coverage	sed -n '603,655p' fastworkflow/command_directory.py
freshness gate	sed -n '384,407p' fastworkflow/command_routing.py
snapshot JSON keys	python -c "import json;print(list(json.load(open('tests/example_workflow/___command_info/command_directory.json'))))"
model artifact filenames per context	ls fastworkflow/_workflows/command_metadata_extraction/___command_info/global/
param_labeled naming	grep -n "param_labeled" fastworkflow/train/__main__.py
CommandRouter decision rule / thresholds	sed -n '289,335p' fastworkflow/model_pipeline_training.py
trained-check semantics	grep -n "def is_workflow_trained" -A 15 fastworkflow/model_pipeline_training.py
action.jsonl lifecycle	grep -n "action.jsonl|mirror_action_log" fastworkflow/workflow_execution_context.py fastworkflow/chat_session.py
trace event emission sites	grep -n "CommandTraceEvent" fastworkflow/workflow_agent.py fastworkflow/workflow_execution_context.py
server trace dict keys	grep -n "_format_trace_event" -A 14 fastworkflow/run_fastapi_mcp/utils.py
turn response body keys	grep -n "def render_turn_response" -A 40 fastworkflow/run_fastapi_mcp/turns.py
destructive drain still unfixed	grep -n "Destructive trace drain" fastworkflow/run_fastapi_mcp/turns.py
LOG_LEVEL reconfigure path	grep -n "reconfigure_log_level" fastworkflow/__init__.py fastworkflow/utils/logging.py
probe endpoints	grep -n "probes/" fastworkflow/run_fastapi_mcp/__main__.py | head
DSPy cache utility	.venv/bin/python -m fastworkflow.utils.dspy_cache_utils status
open-issue statuses (fix-85g, fix-5fv)	bd show fix-85g; bd show fix-5fv (read-only)
version/commit baseline	grep -n '^version' pyproject.toml; git log --oneline -1