08-debugging

# Track A Step 8: Debugging Deployed Agents Systematically diagnose and resolve issues with agents deployed to Databricks Apps. > **Source documentation:** > This skill is grounded in [Debug a deployed AI agent](https://docs.databricks.com/aws/en/generative-ai/agent-framework/debug-agent) > (Databricks docs). The source page covers Apps **and** Model Serving > debugging and is updated as the platform evolves. **If a command, API, or > error message in this skill does not match what you see, consult the source > page first** — it is the canonical reference. Related pages: > [Deploy a Databricks App](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/deploy), > [Add resources to a Databricks app](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources). ## When to Use - Your deployed app returns errors, 302 redirects, or 502s. - The agent responds but ignores tools, hallucinates, or drops context. - Resource permission errors appear in logs. - Lakebase memory is not persisting across conversations. - You need to validate configuration before a deploy. - Local dev server (`uv run start-app`) is failing. --- ## Best Practices Follow these before you start debugging — they prevent most issues: 1. **Enable MLflow tracing.** `mlflow.openai.autolog()` at module level (configured in A2). Traces are the single most useful diagnostic tool. 2. **Document tools clearly.** Clear tool and parameter descriptions ensure the LLM calls tools correctly. See A3 for `@function_tool` docstring patterns. 3. **Add timeouts and token limits to LLM calls.** Prevents delays from long-running steps. If your agent uses the OpenAI client to query a Databricks serving endpoint, set custom timeouts on the calls. 4. **Validate configuration before deployment.** Run `databricks bundle validate` before `databricks bundle deploy` to catch YAML issues early. 5. **Test locally first.** Use `uv run start-app` to catch issues before deploying. Send test requests, verify traces appear in MLflow, then deploy. --- ## Debug Local Development Before deploying, verify your local environment is configured correctly. ### Environment checklist ```bash # 1. Check Databricks CLI version (need 0.283.0+) databricks -v # 2. Verify authentication profiles databricks auth profiles # 3. Verify .env contains MLFLOW_TRACKING_URI in correct format grep MLFLOW_TRACKING_URI .env # Must be: databricks://PROFILE_NAME (not a URL) ``` ### Common local development errors | Error | Cause | Fix | |-------|-------|-----| | `The provided MLFLOW_EXPERIMENT_ID does not exist` | Wrong tracking URI format or experiment deleted | Verify `MLFLOW_TRACKING_URI` uses `databricks://PROFILE_NAME` format | | `ModuleNotFoundError` on start | Dependencies not installed | Run `uv sync` to install dependencies | | Port 8000 already in use | Another process on the port | `lsof -ti:8000 | xargs kill` or use `uv run start-app --port 8001` | | Authentication errors locally | Environment not configured | Run `uv run quickstart` or manually configure `.env` | ### Test the agent locally ```bash # Terminal 1: Start the agent server uv run start-app # Terminal 2: Send a test request curl -X POST http://localhost:8000/invocations \ -H "Content-Type: application/json" \ -d '{"input": [{"role": "user", "content": "hello"}]}' ``` View MLflow traces in the Databricks UI to verify your agent is logging traces correctly. If the server starts but returns no useful reply, check: 1. The server terminal for tracebacks or HTTP errors. 2. `.env` is populated — compare keys with `.env.example`. 3. `databricks auth token` succeeds for the profile quickstart used. --- ## Debug Configuration Configuration errors in `databricks.yml` and `app.yaml` are the most common source of deployment failures. ### Validate before deploying > **Client note:** IDE runs this in a terminal; Genie Code runs the `databricks bundle …` command via `runDatabricksCli` (be on the bundle's page). See `skills/genie-code-environment`. ```bash databricks bundle validate ``` This catches YAML syntax errors, missing required fields, invalid resource references, and permission configuration issues. ### Common configuration mismatches | Configuration Point | Rule | How to Debug | |---------------------|------|-------------| | `valueFrom` references in `app.yaml` | Must exactly match a resource name in `databricks.yml` | Search for the exact string in both files | | App name | Must start with `agent-` prefix | Check the `name` field under `resources.apps` in `databricks.yml` | | Genie space ID | Must be the 32-character hex string from the URL | Extract from `https://...cloud.databricks.com/genie/rooms/{SPACE_ID}` | | Unity Catalog function reference | Must use `catalog.schema.function_name` format | Verify with `databricks unity-catalog functions list` | | Lakebase instance reference | Must use `value` (not `valueFrom`) in `app.yaml` | The instance name is a literal string, not a resource reference | ### Example: Spotting a valueFrom mismatch ```yaml # app.yaml env: - name: SQL_WAREHOUSE_ID valueFrom: sql-warehouse # <-- Must match name below # databricks.yml resources: apps: my_agent: resources: - name: sql-warehouse # <-- This must match sql_warehouse: id: "abc123" permission: CAN_USE ``` If `valueFrom` says `sql_warehouse` but the resource name is `sql-warehouse`, deployment silently fails to inject the environment variable. --- ## Debug Deployment ### App already exists error If you see `Error: failed to create app - An app with the same name already exists`: **Option 1: Bind to the existing app (recommended)** ```bash databricks apps get <app-name> --output json databricks bundle deployment bind <bundle-name> <app-name> --auto-approve databricks bundle deploy databricks bundle run <bundle-name> ``` **Option 2: Delete and recreate** ```bash databricks apps delete <app-name> databricks bundle deploy databricks bundle run <bundle-name> ``` ### App not updating after deploy `databricks bundle deploy` only uploads files to the workspace. You must also run `databricks bundle run <bundle-name>` to restart the app with the new code. Always deploy using both commands: ```bash databricks bundle deploy && databricks bundle run <bundle-name> ``` ### View deployment status and logs ```bash # Check app status databricks apps get <app-name> # View real-time logs databricks apps logs <app-name> --follow ``` Look for stack traces, permission denied messages, connection errors, and timeout messages in the log output. --- ## Debug Runtime Errors ### Analyze app logs ```bash databricks apps logs <app-name> --follow ``` Look for: - Stack traces indicating code errors - `Permission denied` messages for resources - Connection errors to external services (MCP servers, serving endpoints) - Timeout messages ### Common runtime errors | Error | Cause | Fix | |-------|-------|-----| | 302 redirect when querying app | Using a PAT instead of OAuth | Get an OAuth token with `databricks auth token` | | Agent not using available tools | Tools not returned from MCP client | Verify the MCP server URL is correct and the resource has proper permissions in `databricks.yml` | | Streaming response breaks mid-response | Connection timeout | Increase `CHAT_PROXY_TIMEOUT_SECONDS` in `app.yaml` env section | | Agent returning "Memory not available" | Missing `user_id` in request | Pass `custom_inputs.user_id` in the request payload | | Empty or error responses despite 200 status | Error within streamed response | Check the actual stream content and app logs, not just the HTTP status code | ### Use MLflow traces for diagnosis When the agent responds but incorrectly, MLflow traces are the primary diagnostic tool: 1. Open your MLflow experiment in the Databricks UI. 2. Find the trace for the failing request. 3. Inspect each span (AGENT, LLM, TOOL) for: - **LLM spans**: Was the prompt correct? Did the model receive the right context? - **TOOL spans**: Did the tool receive the right arguments? Did it return the expected result? Did it error? - **AGENT spans**: Did the orchestration route correctly? --- ## Debug Authentication ### OAuth token requirement Databricks Apps require OAuth tokens. Personal Access Tokens (PATs) result in a 302 redirect. ```bash # Get an OAuth token databricks auth token # Use it in requests TOKEN=$(databricks auth token | jq -r '.access_token') curl -X POST <app-url>/invocations \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"input": [{"role": "user", "content": "hello"}]}' ``` ### Resource permission errors When the agent cannot access workspace resources, verify the resource is configured in `databricks.yml`. Each resource type requires specific permissions: | Error | Cause | Fix | |-------|-------|-----| | Permission denied on Genie space | Missing `genie_space` resource | Add `genie_space` with `permission: 'CAN_RUN'` | | Vector search index not accessible | Missing `uc_securable` for the index | Add `uc_securable` with `securable_type: 'TABLE'`, `permission: 'SELECT'` | | UC function execution denied | Missing `uc_securable` for the function | Add `uc_securable` with `securable_type: 'FUNCTION'`, `permission: 'EXECUTE'` | | Serving endpoint access denied | Missing `serving_endpoint` resource | Add `serving_endpoint` with `permission: 'CAN_QUERY'` | | SQL warehouse access denied | Missing `sql_warehouse` resource | Add `sql_warehouse` with `permission: 'CAN_USE'` | Example resource configuration in `databricks.yml`: ```yaml resources: apps: my_agent: name: 'agent-my-app' resources: - name: 'my_genie_space' genie_space: space_id: '01234567890abcdef01234567890abcd' permission: 'CAN_RUN' - name: 'my_vector_index' uc_securable: securable_full_name: 'catalog.schema.index_name' securable_type: 'TABLE' permission: 'SELECT' ``` ### Custom MCP server permissions If your agent connects to a custom MCP server running as a Databricks app, grant permissions manually (apps are not yet supported as resource dependencies in `databricks.yml`): ```bash # Get your agent app's service principal AGENT_SP=$(databricks apps get <agent-app-name> --output json | jq -r '.service_principal_name') # Grant permission on the MCP server app databricks apps update-permissions <mcp-server-app-name> \ --json "{\"access_control_list\": [{\"service_principal_name\": \"$AGENT_SP\", \"permission_level\": \"CAN_USE\"}]}" ``` --- ## Debug Lakebase Memory For agents using Lakebase for memory storage (configured in A5): | Error | Cause | Fix | |-------|-------|-----| | `relation 'store' does not exist` | Memory tables not initialized | Run `await store.setup()` locally before deploying | | `Unable to resolve :re[LKB] instance` | Wrong instance name or configuration | Verify `LAKEBASE_INSTANCE_NAME` uses `value` (not `valueFrom`) in `app.yaml` and matches the `instance_name` in `databricks.yml` | | `permission denied for table store` | Missing Lakebase permissions | Add a `database` resource in `databricks.yml` with `permission: 'CAN_CONNECT_AND_CREATE'` | | Memory not persisting across conversations | Different `user_id` per request | Pass a consistent `user_id` in `custom_inputs` for each user | ### Initialize tables before deploying ```python import asyncio from databricks_langchain import AsyncDatabricksStore async def setup_memory(): async with AsyncDatabricksStore( instance_name='your-lakebase-instance', embedding_endpoint='databricks-gte-large-en', embedding_dims=1024, ) as store: await store.setup() asyncio.run(setup_memory()) ``` ### Lakebase resource configuration ```yaml resources: apps: my_agent: resources: - name: 'memory_database' database: instance_name: '<lakebase-instance-name>' database_name: 'postgres' permission: 'CAN_CONNECT_AND_CREATE' ``` --- ## Agent-as-Judge Debugging When an agent misbehaves on specific traces, raw span-tree inspection is slow. **Agent-as-judge** uses an LLM judge to read a trace and explain — in natural language — *why* it failed against your guidelines. This is faster than eyeballing 30 spans and often surfaces root causes humans miss. ### When to reach for agent-as-judge - You have ≥ 5 failing production traces in `*_otel_traces` but can't see a pattern. - A scorer is firing below threshold and you can't tell which span caused it. - A user complaint points to a specific `request_id` and you need a fast triage narrative. - You want to auto-generate "debug notes" for every failing CI eval row. Don't use when: - You already know the failure is a tool error (check the span status directly). - You have < 3 failing rows — read them by hand. ### Using `make_judge` for trace-level failure analysis ```python from mlflow.genai import make_judge import mlflow failure_judge = make_judge( name="agent_failure_root_cause", instructions=( "Read the trace. Identify the FIRST step where the agent deviated from the " "expected behavior per SkyLoyalty policies. Answer in this structured form:\n" " * Failure step: <name of the span or tool call>\n" " * Root cause: <one sentence>\n" " * Evidence: <quote 1-2 lines from the span>\n" " * Recommended fix: <one action>\n" "If the trace succeeded, respond only with 'OK'." ), model="databricks:/databricks-claude-sonnet-4-6", ) trace = mlflow.get_trace("<request_id>") feedback = failure_judge(trace=trace) print(feedback.value) # 'OK' or the structured diagnosis print(feedback.rationale) # Model's full reasoning ``` `make_judge` accepts a `trace` argument — the judge reads span names, inputs, outputs, and attributes directly. No manual extraction needed. ### Batching across failing traces Run the judge against the failing subset from [07-production-monitoring](../../../sdlc/07-production-monitoring/SKILL.md) and cluster the diagnoses: ```python from databricks.sdk import WorkspaceClient import mlflow import collections w = WorkspaceClient() failing_ids = w.sql.query(""" SELECT DISTINCT request_id FROM main.skyloyalty_ops.skyloyalty_agent_otel_annotations