07-deploy-and-query

# Track A Step 7: Deploy and Query Deploy your evaluated agent to Databricks Apps and verify it works in production. ## When to Use - You have a working, evaluated agent from Steps 1-6. - You are ready to deploy to Databricks Apps for team/production use. - You need to query the deployed agent programmatically. ## Relationship to SDLC Deployment This step does a **first manual deployment** for testing. The SDLC deployment pipeline ([S5](../../../sdlc/05-logged-model-and-uc-registration/SKILL.md)–[S6](../../../sdlc/06-deployment-and-automation/SKILL.md)) adds model registration in Unity Catalog, champion gating, Databricks Asset Bundles, and evaluate-then-promote CI/CD automation. Think of A7 as "get it running" and S6 as "make it production-grade." | Aspect | A7 (this step) | S6 (SDLC) | |--------|----------------|-----------| | **Scope** | First manual deploy | Automated CI/CD pipeline | | **Method** | `databricks apps deploy` CLI | Databricks Asset Bundles + CI/CD | | **Gating** | Manual eval check (A6) | Automated eval threshold gate (S4) | | **Model registry** | None | UC-registered model with champion/challenger | | **Monitoring** | Manual trace inspection | [S7: Production Monitoring](../../../sdlc/07-production-monitoring/SKILL.md) | After completing Track A (A1-A7), proceed to the [SDLC pipeline](../../../sdlc/01-prompt-registry/SKILL.md) to productionize your agent. --- ## Two Deployment Paths | Path | When to Use | |---|---| | **Workspace UI** | First-time setup, want the guided experience | | **CLI** | CI/CD, repeat deployments, scripted workflows | Both produce the same result — a running Databricks App serving your agent. --- ## Path A: Deploy via Workspace UI 1. In your Databricks workspace, click **+ New > App**. 2. Select **Agents > Agent - OpenAI Agents SDK**. 3. Create a new MLflow experiment (e.g., `openai-agents-template`). 4. Complete the setup wizard — the app deploys automatically. 5. Click the **App URL** to open the built-in chat UI. After creating the app in the UI, download the source to your local machine for customization: ```bash # Copy the sync command from the app's page databricks sync --watch . "/Users/<your-username>/agent-openai-agents-sdk" ``` This is the easiest path for first-time deployment. For subsequent updates, use the CLI workflow below. --- ## Path B: Deploy via CLI > **Genie Code:** run the create / sync / deploy CLI through `runDatabricksCli` (pre-authenticated) and be on the app's page; the enhanced CLI build flow is page/CWD-dependent, so the SDK `w.apps.deploy(...)` SNAPSHOT path is the reliable fallback, and verify the deployed app via the OAuth session. See `skills/genie-code-environment` §3–§4, §7. ### Step 7b-1: Create the App If you cloned the repository locally (not via the Workspace UI), create the app first: ```bash databricks apps create agent-openai-agents-sdk ``` Replace `agent-openai-agents-sdk` with your app name. If you already created the app via the Workspace UI, skip this step. ### Step 7b-2: Sync Files to Workspace Upload your local project files to the Databricks workspace: ```bash DATABRICKS_USERNAME=$(databricks current-user me | jq -r .userName) databricks sync . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk" ``` This copies your project directory to a workspace folder under your user namespace. The `databricks sync` command uploads all files that are not excluded by `.gitignore` or `.databricksignore`. ### Step 7b-3: Deploy the App ```bash databricks apps deploy agent-openai-agents-sdk \ --source-code-path "/Workspace/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk" ``` The deploy command: 1. Reads `app.yaml` for compute size and resource declarations 2. Installs dependencies from `pyproject.toml` 3. Starts the agent on the configured compute 4. Makes the app available at `<app-name>.<workspace>.databricksapps.com` ### Step 7b-4: Verify Deployment ```bash # Check app status databricks apps get agent-openai-agents-sdk # Get the app URL databricks apps get agent-openai-agents-sdk | jq -r .url ``` Open the app URL in a browser — the built-in chat UI should load and respond to messages. --- ## Redeployment Workflow When you make code changes, re-sync and redeploy: ```bash # 1. Sync updated files databricks sync . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk" # 2. Redeploy (picks up new files) databricks apps deploy agent-openai-agents-sdk \ --source-code-path "/Workspace/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk" ``` For iterative development, use `--watch` with sync to auto-upload on save: ```bash databricks sync --watch . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk" ``` Then redeploy when ready to pick up the changes. --- ## Querying the Deployed Agent ### Authentication: OAuth Tokens (PATs Do Not Work) Databricks Apps require **OAuth tokens** — Personal Access Tokens (PATs) are **not supported**. Generate an OAuth token: ```bash # Authenticate once — IDE/CLI only (see PRE-REQUISITES §11). Genie Code: pre-authenticated, run via runDatabricksCli. # Get a fresh token databricks auth token --host https://<workspace>.cloud.databricks.com ``` ### Query with curl ```bash APP_URL="https://agent-openai-agents-sdk.<workspace>.databricksapps.com" TOKEN=$(databricks auth token --host https://<workspace>.cloud.databricks.com | jq -r .access_token) # Non-streaming curl -X POST "$APP_URL/invocations" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"input": [{"role": "user", "content": "Hello, what can you do?"}]}' # Streaming curl -X POST "$APP_URL/invocations" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"input": [{"role": "user", "content": "Hello"}], "stream": true}' ``` ### Query with Python (requests) ```python import subprocess import json import requests token_output = subprocess.run( ["databricks", "auth", "token", "--host", "https://<workspace>.cloud.databricks.com"], capture_output=True, text=True ) token = json.loads(token_output.stdout)["access_token"] app_url = "https://agent-openai-agents-sdk.<workspace>.databricksapps.com" response = requests.post( f"{app_url}/invocations", headers={ "Authorization": f"Bearer {token}", "Content-Type": "application/json", }, json={ "input": [{"role": "user", "content": "What can you help me with?"}], }, ) print(response.json()) ``` ### Query with the Databricks OpenAI Client For the most ergonomic Python experience, use the OpenAI-compatible client: ```python from databricks.sdk import WorkspaceClient w = WorkspaceClient() openai_client = w.serving_endpoints.get_open_ai_client() response = openai_client.responses.create( model="agent-openai-agents-sdk", input=[{"role": "user", "content": "Hello"}], ) print(response.output_text) ``` For the full list of query methods, see [Query an agent deployed on Databricks](https://docs.databricks.com/aws/en/generative-ai/agent-framework/query-agent). --- ## Sending custom_inputs to Deployed Agents ```bash curl -X POST "$APP_URL/invocations" \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{ "input": [{"role": "user", "content": "What were Q3 sales?"}], "custom_inputs": {"user_id": "analyst@acme.com", "session_id": "s-123"} }' ``` See `02-agent-framework/references/custom-inputs-outputs.md` for the full `custom_inputs` / `custom_outputs` pattern. --- ## Pre-deploy: verify F2 grants and Lakebase policies A7 is the **consumer** of two upstream contracts; it does not derive either at deploy time. ### F2 (`02-experiment-tracing-and-uc-storage`) state must be present Before running `databricks apps deploy`, verify the four F2 fields are captured in state and `f2_grants_complete` is `true`. The deploy script should fail closed if any of these are missing: ```python state = load_state() assert state["f2_grants_complete"] is True, ( "Refusing to deploy: f2_grants_complete is not true. " "Run F2 (foundation/02-experiment-tracing-and-uc-storage) and apply the " "MODIFY+SELECT grants on every *_otel_* table to the app SP, then capture " "app_service_principal_grants[] before retrying." ) assert state.get("otel_table_prefix"), "Missing otel_table_prefix (F2-owned)" assert state.get("mlflow_tracing_sql_warehouse_id"), ( "Missing mlflow_tracing_sql_warehouse_id — preflight_check_registry." "mlflow_tracing_sql_warehouse_id_present blocks deploy." ) ``` ### Lakebase policies (if the agent uses Lakebase memory) must be present If the agent uses Lakebase memory (Track A 05 / `agent-openai-advanced`), A2 (`02-agent-framework`) must have captured the Lakebase client policy in state: ```python if state.get("uses_lakebase"): assert state.get("lakebase_endpoint_uri") policy = state.get("lakebase_cold_start_retry_policy") or {} assert "AdminShutdown" in policy.get("retry_on", []) assert "psycopg_pool.PoolClosed" in policy.get("retry_on", []) assert policy.get("max_attempts", 0) >= 1, ( "preflight_check_registry.lakebase_cold_start_retry_policy_present " "fails — set max_attempts >= 1 in A2 before deploy." ) assert state.get("lakebase_pool_close_policy") ``` The deploy script also greps the agent module for forbidden module-level Lakebase init: ```bash # fail closed if the agent opens a Lakebase pool at import time if grep -nE '^[^#].*AsyncConnectionPool\(.*open=True' agent.py >/dev/null; then echo "Refusing to deploy: Lakebase pool opened at module import. Move into handle_invoke/handle_stream." exit 1 fi ``` --- ## Production tracing checklist Track A deploys agents as **Databricks Apps**, so the **Agent Framework auto-tracing** path applies — but the canonical env-var matrix and the **Git-folder caveat** are the same as for custom serving and apply here too. Before declaring the deploy "done": - [ ] Experiment is in a **non-Git** path. Move it to `/Shared/...` if it currently lives under a Databricks Git folder, or trace ingestion will silently drop. - [ ] `mlflow.set_experiment(...)` to that non-Git path is called **before** deploy. - [ ] App service principal has **`CAN_EDIT`** on the experiment (Experiment → Permissions). Without it, traces 403 on first request. - [ ] `MLFLOW_EXPERIMENT_ID` set in `app.yaml` (or `databricks.yml` env) so the deployed runtime knows where to write. - [ ] Send one production request after deploy and confirm a trace appears in the **MLflow Experiment UI** (with token counts and `gen_ai.*` attrs if `mlflow.openai.autolog()` is enabled at module level). - [ ] (UC OTEL) `MLFLOW_TRACING_SQL_WAREHOUSE_ID` set so traces also land in Delta tables for SQL querying. For the full env-var matrix, the Git-folder caveat in detail, the custom-CPU-serving variant, and the Production-Monitoring-to-Delta pattern, see the canonical reference: - [`foundation/02-experiment-tracing-and-uc-storage/references/prod-tracing-deployment.md`](../../../foundation/02-experiment-tracing-and-uc-storage/references/prod-tracing-deployment.md) --- ## DO / DON'T ### DO — Evaluate before every deploy ```bash uv run agent-evaluate # Step 6 # Only if evaluation passes: databricks sync . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk" databricks apps deploy agent-openai-agents-sdk --source-code-path ... ``` ### DON'T — Deploy without syncing first ```bash # Missing sync — deploys stale code! databricks apps deploy agent-openai-agents-sdk --source-code-path ... ``` ### DO — Use OAuth tokens for Apps ```bash TOKEN=$(databricks auth token --host ... | jq -r .access_token) curl -H "Authorization: Bearer $TOKEN" "$APP_URL/invocations" ... ```