07-deploy-and-query

name	07-deploy-and-query
description	Use when deploying an agent to Databricks Apps and querying the deployed endpoint. Covers databricks apps create, sync, deploy, OAuth token auth, PAT limitations, curl and Python SDK query methods, and the redeploy workflow. Track A Step 7. Consumes evaluation results from Step 6. Produces a running deployed agent on Databricks Apps.
license	Apache-2.0
clients	["ide_cli","genie_code"]
bundle_resource	apps
deploy_verb	apps_deploy
deploy_note	The Track A deploy step — create/sync/deploy the agent to the Databricks Apps runtime, then query the endpoint via OAuth. On Genie Code run the create/sync/deploy CLI through runDatabricksCli (pre-authenticated); on IDE via the local CLI. The deployed app + endpoint carry the per-user prefix. See `skills/genie-code-environment` for the resolved deploy verbs.
coverage	full
metadata	{"last_verified":"2026-06-05","volatility":"high","upstream_sources":[],"author":"prashanth-subrahmanyam","version":"1.1.0","domain":"genai-agents","pipeline_position":"A7","consumes":"evaluation_results, authenticated_client, agent_with_tools, f2_grants_complete, otel_table_prefix, mlflow_tracing_sql_warehouse_id, app_service_principal_grants, lakebase_endpoint_uri, lakebase_cold_start_retry_policy, lakebase_pool_close_policy","produces":"deployed_app, app_url","grounded_in":"docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent#step-7-deploy-the-agent-to-databricks-apps, docs.databricks.com/aws/en/generative-ai/agent-framework/author-agent#step-8-query-the-deployed-agent, docs.databricks.com/aws/en/dev-tools/databricks-apps/deploy"}

Track A Step 7: Deploy and Query

Deploy your evaluated agent to Databricks Apps and verify it works in production.

When to Use

• You have a working, evaluated agent from Steps 1-6.
• You are ready to deploy to Databricks Apps for team/production use.
• You need to query the deployed agent programmatically.

Relationship to SDLC Deployment

This step does a first manual deployment for testing. The SDLC deployment pipeline (S5–S6) adds model registration in Unity Catalog, champion gating, Databricks Asset Bundles, and evaluate-then-promote CI/CD automation. Think of A7 as "get it running" and S6 as "make it production-grade."

Aspect	A7 (this step)	S6 (SDLC)
Scope	First manual deploy	Automated CI/CD pipeline
Method	databricks apps deploy CLI	Databricks Asset Bundles + CI/CD
Gating	Manual eval check (A6)	Automated eval threshold gate (S4)
Model registry	None	UC-registered model with champion/challenger
Monitoring	Manual trace inspection	S7: Production Monitoring

After completing Track A (A1-A7), proceed to the SDLC pipeline to productionize your agent.

---

Two Deployment Paths

Path	When to Use
Workspace UI	First-time setup, want the guided experience
CLI	CI/CD, repeat deployments, scripted workflows

Both produce the same result — a running Databricks App serving your agent.

---

Path A: Deploy via Workspace UI

1. In your Databricks workspace, click + New > App.
2. Select Agents > Agent - OpenAI Agents SDK.
3. Create a new MLflow experiment (e.g., openai-agents-template).
4. Complete the setup wizard — the app deploys automatically.
5. Click the App URL to open the built-in chat UI.

After creating the app in the UI, download the source to your local machine for customization:

# Copy the sync command from the app's page
databricks sync --watch . "/Users/<your-username>/agent-openai-agents-sdk"

This is the easiest path for first-time deployment. For subsequent updates, use the CLI workflow below.

---

Path B: Deploy via CLI

Genie Code: run the create / sync / deploy CLI through runDatabricksCli (pre-authenticated) and be on the app's page; the enhanced CLI build flow is page/CWD-dependent, so the SDK w.apps.deploy(...) SNAPSHOT path is the reliable fallback, and verify the deployed app via the OAuth session. See skills/genie-code-environment §3–§4, §7.

Step 7b-1: Create the App

If you cloned the repository locally (not via the Workspace UI), create the app first:

databricks apps create agent-openai-agents-sdk

Replace agent-openai-agents-sdk with your app name. If you already created the app via the Workspace UI, skip this step.

Step 7b-2: Sync Files to Workspace

Upload your local project files to the Databricks workspace:

DATABRICKS_USERNAME=$(databricks current-user me | jq -r .userName)
databricks sync . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk"

This copies your project directory to a workspace folder under your user namespace. The databricks sync command uploads all files that are not excluded by .gitignore or .databricksignore.

Step 7b-3: Deploy the App

databricks apps deploy agent-openai-agents-sdk \
  --source-code-path "/Workspace/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk"

The deploy command:

1. Reads app.yaml for compute size and resource declarations
2. Installs dependencies from pyproject.toml
3. Starts the agent on the configured compute
4. Makes the app available at <app-name>.<workspace>.databricksapps.com

Step 7b-4: Verify Deployment

# Check app status
databricks apps get agent-openai-agents-sdk

# Get the app URL
databricks apps get agent-openai-agents-sdk | jq -r .url

Open the app URL in a browser — the built-in chat UI should load and respond to messages.

---

Redeployment Workflow

When you make code changes, re-sync and redeploy:

# 1. Sync updated files
databricks sync . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk"

# 2. Redeploy (picks up new files)
databricks apps deploy agent-openai-agents-sdk \
  --source-code-path "/Workspace/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk"

For iterative development, use --watch with sync to auto-upload on save:

databricks sync --watch . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk"

Then redeploy when ready to pick up the changes.

---

Querying the Deployed Agent

Authentication: OAuth Tokens (PATs Do Not Work)

Databricks Apps require OAuth tokens — Personal Access Tokens (PATs) are not supported.

Generate an OAuth token:

# Authenticate once — IDE/CLI only (see PRE-REQUISITES §11). Genie Code: pre-authenticated, run via runDatabricksCli.

# Get a fresh token
databricks auth token --host https://<workspace>.cloud.databricks.com

Query with curl

APP_URL="https://agent-openai-agents-sdk.<workspace>.databricksapps.com"
TOKEN=$(databricks auth token --host https://<workspace>.cloud.databricks.com | jq -r .access_token)

# Non-streaming
curl -X POST "$APP_URL/invocations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": [{"role": "user", "content": "Hello, what can you do?"}]}'

# Streaming
curl -X POST "$APP_URL/invocations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input": [{"role": "user", "content": "Hello"}], "stream": true}'

Query with Python (requests)

import subprocess
import json
import requests

token_output = subprocess.run(
    ["databricks", "auth", "token", "--host", "https://<workspace>.cloud.databricks.com"],
    capture_output=True, text=True
)
token = json.loads(token_output.stdout)["access_token"]

app_url = "https://agent-openai-agents-sdk.<workspace>.databricksapps.com"

response = requests.post(
    f"{app_url}/invocations",
    headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json",
    },
    json={
        "input": [{"role": "user", "content": "What can you help me with?"}],
    },
)

print(response.json())

Query with the Databricks OpenAI Client

For the most ergonomic Python experience, use the OpenAI-compatible client:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()
openai_client = w.serving_endpoints.get_open_ai_client()

response = openai_client.responses.create(
    model="agent-openai-agents-sdk",
    input=[{"role": "user", "content": "Hello"}],
)
print(response.output_text)

For the full list of query methods, see Query an agent deployed on Databricks.

---

Sending custom_inputs to Deployed Agents

curl -X POST "$APP_URL/invocations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "input": [{"role": "user", "content": "What were Q3 sales?"}],
    "custom_inputs": {"user_id": "[e-mail]", "session_id": "s-123"}
  }'

See 02-agent-framework/references/custom-inputs-outputs.md for the full custom_inputs / custom_outputs pattern.

---

Pre-deploy: verify F2 grants and Lakebase policies

A7 is the consumer of two upstream contracts; it does not derive either at deploy time.

F2 (02-experiment-tracing-and-uc-storage) state must be present

Before running databricks apps deploy, verify the four F2 fields are captured in state and f2_grants_complete is true. The deploy script should fail closed if any of these are missing:

state = load_state()
assert state["f2_grants_complete"] is True, (
    "Refusing to deploy: f2_grants_complete is not true. "
    "Run F2 (foundation/02-experiment-tracing-and-uc-storage) and apply the "
    "MODIFY+SELECT grants on every *_otel_* table to the app SP, then capture "
    "app_service_principal_grants[] before retrying."
)
assert state.get("otel_table_prefix"), "Missing otel_table_prefix (F2-owned)"
assert state.get("mlflow_tracing_sql_warehouse_id"), (
    "Missing mlflow_tracing_sql_warehouse_id — preflight_check_registry."
    "mlflow_tracing_sql_warehouse_id_present blocks deploy."
)

Lakebase policies (if the agent uses Lakebase memory) must be present

If the agent uses Lakebase memory (Track A 05 / agent-openai-advanced), A2 (02-agent-framework) must have captured the Lakebase client policy in state:

if state.get("uses_lakebase"):
    assert state.get("lakebase_endpoint_uri")
    policy = state.get("lakebase_cold_start_retry_policy") or {}
    assert "AdminShutdown" in policy.get("retry_on", [])
    assert "psycopg_pool.PoolClosed" in policy.get("retry_on", [])
    assert policy.get("max_attempts", 0) >= 1, (
        "preflight_check_registry.lakebase_cold_start_retry_policy_present "
        "fails — set max_attempts >= 1 in A2 before deploy."
    )
    assert state.get("lakebase_pool_close_policy")

The deploy script also greps the agent module for forbidden module-level Lakebase init:

# fail closed if the agent opens a Lakebase pool at import time
if grep -nE '^[^#].*AsyncConnectionPool\(.*open=True' agent.py >/dev/null; then
  echo "Refusing to deploy: Lakebase pool opened at module import. Move into handle_invoke/handle_stream."
  exit 1
fi

---

Production tracing checklist

Track A deploys agents as Databricks Apps, so the Agent Framework auto-tracing path applies — but the canonical env-var matrix and the Git-folder caveat are the same as for custom serving and apply here too.

Before declaring the deploy "done":

• Experiment is in a non-Git path. Move it to /Shared/... if it currently lives under a Databricks Git folder, or trace ingestion will silently drop.
• mlflow.set_experiment(...) to that non-Git path is called before deploy.
• App service principal has CAN_EDIT on the experiment (Experiment → Permissions). Without it, traces 403 on first request.
• MLFLOW_EXPERIMENT_ID set in app.yaml (or databricks.yml env) so the deployed runtime knows where to write.
• Send one production request after deploy and confirm a trace appears in the MLflow Experiment UI (with token counts and gen_ai.* attrs if mlflow.openai.autolog() is enabled at module level).
• (UC OTEL) MLFLOW_TRACING_SQL_WAREHOUSE_ID set so traces also land in Delta tables for SQL querying.

For the full env-var matrix, the Git-folder caveat in detail, the custom-CPU-serving variant, and the Production-Monitoring-to-Delta pattern, see the canonical reference:

• foundation/02-experiment-tracing-and-uc-storage/references/prod-tracing-deployment.md

---

DO / DON'T

DO — Evaluate before every deploy

uv run agent-evaluate   # Step 6
# Only if evaluation passes:
databricks sync . "/Users/$DATABRICKS_USERNAME/agent-openai-agents-sdk"
databricks apps deploy agent-openai-agents-sdk --source-code-path ...

DON'T — Deploy without syncing first

# Missing sync — deploys stale code!
databricks apps deploy agent-openai-agents-sdk --source-code-path ...

DO — Use OAuth tokens for Apps

TOKEN=$(databricks auth token --host ... | jq -r .access_token)
curl -H "Authorization: Bearer $TOKEN" "$APP_URL/invocations" ...