06-deployment-and-automation

# Deployment and Automation Deploy your agent to **Databricks Apps** (a common production target) and automate **evaluate → gate → promote → deploy** using Databricks Asset Bundles (DAB) and your CI system. ## Upstream Lineage This skill references Databricks Agent Skills' `databricks-mlflow-evaluation` skill for evaluate-gate-promote automation, production trace linkage, and monitoring handoff guidance. If release gates depend on eval harness semantics or upstream monitoring patterns, consult the upstream skill first, then apply this skill's Databricks Apps and Asset Bundle deployment contracts. ## When to Use - Evaluation gates passed (SDLC Step 4) and the model is registered in Unity Catalog (Step 5). - You need a repeatable bundle workflow (`databricks bundle deploy` / `run`), not ad-hoc workspace copies. - You want MCP tool connectivity, Supervisor-style orchestration, or production traces linked to app versions. - You need service principal access patterns and post-deploy verification. --- ## Versioned Resource Path Contract Every persisted workshop artifact that can be superseded must be versioned. The CI/CD pipeline owned by this skill **reads** three of these and **emits** the fourth: - signoffs (read): `/Volumes/<catalog>/<schema>/signoffs/v<N>/decision.md` — owned by [`04b-stakeholder-signoff`](../04b-stakeholder-signoff/SKILL.md) - eval summaries (read): `/Volumes/<catalog>/<schema>/eval_runs/v<N>/summary.json` — owned by [`04-evaluation-runs`](../04-evaluation-runs/SKILL.md) - prompt candidates (read): `prompts:/{catalog}.{uc_agent_schema}.system_instructions@candidate_v<N>` — owned by [`01-prompt-registry`](../01-prompt-registry/SKILL.md) / [`08b-prompt-handauthoring`](../08b-prompt-handauthoring/SKILL.md) - deployment plans (emit): `/Volumes/<catalog>/<schema>/deployment_plans/v<N>/plan.md` `<N>` is a monotonically increasing integer per artifact type, scoped to `(catalog, schema, artifact_type)` (or `(catalog, uc_agent_schema, prompt_name)` for prompt candidates). Resolve `<catalog>` and `<schema>` from state — never hard-code. The promote/deploy step MUST verify that the **same `<N>`** is referenced consistently across the eval summary, signoff decision, and prompt candidate before promoting; releasing against a mixed set of versions silently breaks rollback. Always write a fresh `v<N+1>` for the deployment plan; never overwrite an existing version. --- ## Production Registration Gate: Structured Signoff Consumption The promote step MUST consume the **structured YAML front matter** in `signoffs/v<N>/decision.md` (owned by [`04b-stakeholder-signoff`](../04b-stakeholder-signoff/SKILL.md)). Substring grep on the markdown body is forbidden — narrative text legitimately mentions words like "APPROVED" or "REJECTED" in past tense or as quoted reasons. The signoff document carries two independent decisions. Both must clear before production registration runs: - `engineering_signoff.decision` — set to `APPROVED` or `APPROVED_WITH_CONDITIONS`. - `stakeholder_signoff.decision` — set to `APPROVED` or `APPROVED_WITH_CONDITIONS`. Any of `REJECTED`, missing block, or unparseable YAML blocks production registration. The single audit-tracked escape hatch is a `state_override` block in the same front matter that **captures the original decision verbatim** so reviewers can reconstruct what was bypassed. ```python # In the CI promote step — runs BEFORE setting the @champion alias # or calling databricks bundle deploy --target prod. from pathlib import Path import yaml ALLOWED = {"APPROVED", "APPROVED_WITH_CONDITIONS"} signoff_path = Path(f"/Volumes/{catalog}/{schema}/signoffs") / f"v{version}" / "decision.md" text = signoff_path.read_text() assert text.startswith("---\n"), "signoff missing YAML front matter" _, front, _ = text.split("---\n", 2) meta = yaml.safe_load(front) or {} eng = (meta.get("engineering_signoff") or {}).get("decision") biz = (meta.get("stakeholder_signoff") or {}).get("decision") if eng not in ALLOWED or biz not in ALLOWED: override = meta.get("state_override") or {} captured_eng = (override.get("engineering_signoff") or {}).get("decision") captured_biz = (override.get("stakeholder_signoff") or {}).get("decision") if captured_eng != eng or captured_biz != biz: raise SystemExit( f"Blocked: engineering={eng}, stakeholder={biz}; no state_override " "captures the original decisions. Production registration aborted." ) # Only here may the pipeline proceed to alias promotion + bundle deploy. ``` This gate runs **before** any of the following actions: 1. Setting the `@champion` (or production) alias in Unity Catalog. 2. Issuing `databricks bundle deploy --target prod`. 3. Updating `MLFLOW_ACTIVE_MODEL_ID` in the production app environment. If the gate raises, the entire promote step exits non-zero and CI surfaces the failure to the requester. There is no retry without first re-running the signoff workflow in [`04b-stakeholder-signoff`](../04b-stakeholder-signoff/SKILL.md). --- ## Databricks Asset Bundles (DAB) > **Genie Code:** run every deploy command through `runDatabricksCli` (pre-authenticated), and be on the bundle's page so the CWD resolves to the bundle root. The CI/CD spine is identical on both clients. See `skills/genie-code-environment` §3–§4. Define jobs, apps, and variables in `databricks.yml`, then deploy and run by target. ```yaml # databricks.yml (illustrative) bundle: name: my_agent_bundle variables: catalog: { default: main } job_id_env: { default: MY_AGENT_JOB_ID } # inject app env from bundle if needed resources: jobs: my_eval_job: name: my-agent-evaluate-and-promote # tasks: notebook / wheel / sql — see DAB docs apps: my_agent_app: name: my-agent-app source_code_path: ./app targets: dev: default: true workspace: host: https://<dev-workspace>.cloud.databricks.com staging: workspace: host: https://<staging-workspace>.cloud.databricks.com prod: workspace: host: https://<prod-workspace>.cloud.databricks.com ``` Workflow: ```bash databricks bundle validate databricks bundle deploy --target dev databricks bundle run --target dev my_agent_app # or job name per your bundle ``` Use **targets** for environment-specific workspace hosts, variables, and overrides (dev / staging / prod). See [Databricks Asset Bundles](https://docs.databricks.com/aws/en/dev-tools/bundles/). --- > **Recommended default:** Deploy to **Databricks Apps** for most agents. Use **Model Serving** only when you need a pure inference endpoint without a custom UI or backend. ## Databricks Apps: `app.yaml` / `app.yml` Declare how the app starts and which platform resources it may use. - **`command`**: process that runs your server (for example `uvicorn` or your framework’s entrypoint). - **`env`**: plain values and bindings. Set **`MLFLOW_ACTIVE_MODEL_ID`** to the UC logged-model identifier (or substitute from bundle variables) so production traces align with the deployed app version — see [Link production traces to app versions](https://docs.databricks.com/aws/en/mlflow3/genai/prompt-version-mgmt/version-tracking/link-production-traces-to-app-versions). - **Tracing env vars**: also set `ENABLE_MLFLOW_TRACING=true`, `MLFLOW_EXPERIMENT_ID=<numeric>`, and `APP_ENVIRONMENT=production` (or `staging`) so traces flow from the deployed runtime and app code can override `mlflow.source.type` via metadata. The full env-var matrix (PAT vs OAuth, SP `CAN_EDIT` requirement, the Git-folder caveat) lives in the canonical reference: [`foundation/02-experiment-tracing-and-uc-storage/references/prod-tracing-deployment.md`](../../foundation/02-experiment-tracing-and-uc-storage/references/prod-tracing-deployment.md). For the `APP_ENVIRONMENT` override pattern and user / session metadata, see [F2c — Trace context and environments](../../foundation/02c-trace-context-and-environments/SKILL.md). - **Resources**: attach SQL warehouse, serving endpoints, MLflow experiment, Lakebase, etc., per [Databricks Apps resources](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/resources). Example fragment: ```yaml command: - "python" - "-m" - "myapp" env: MLFLOW_ACTIVE_MODEL_ID: "{{ logged_model_id }}" # resolve via bundle / CI # Optional: tracing warehouse, feature flags, etc. # resources: (warehouse, endpoints, lakebase — per product docs) ``` Replace `{{ logged_model_id }}` with your bundle variable or CI-injected value (for example the model version URI or ID your org uses). --- ## How `app.yaml` and `databricks.yml` Interact These two files serve different purposes and are read at different times: | File | Read by | When | Purpose | |------|---------|------|---------| | `app.yaml` | Apps platform | App process startup | Runtime config: command, env vars, resource bindings | | `databricks.yml` | `databricks bundle` CLI | Deploy time | Provisioning: create/update app, experiments, jobs, permissions | When deploying via bundles, the `config` block inside `databricks.yml` (`resources.apps.<name>.config`) **overrides** the corresponding fields in `app.yaml`. Specifically: - `config.command` in the bundle replaces `command` in `app.yaml` - `config.env` in the bundle replaces `env` in `app.yaml` - `resources` in the bundle app block replaces `resources` in `app.yaml` **Recommendation:** Use `databricks.yml` as the **source of truth** for all deployment configuration. Keep `app.yaml` as a minimal runtime fallback for local dev or standalone (non-bundle) deploys. Do **not** maintain the same env vars or resource IDs in both files — they will drift and cause confusing deployment mismatches. **Common mistake:** Editing `app.yaml` to fix a deployed app's config, then wondering why `databricks bundle deploy` reverts the change. The bundle always writes its own `config` block. --- ## Preflight (Generic) Before `databricks bundle deploy`, verify: - Databricks CLI auth for the intended workspace / target. - Unity Catalog objects referenced by the app exist (catalogs, schemas, tables, functions). - The app’s service principal (or run-as identity) has required privileges on warehouses, catalogs, and serving endpoints. - Attached serving endpoints respond (health / smoke inference if applicable). Automate these checks in a small script or a DAB job step; do not assume deploy alone validates runtime access. --- ## Service Principal Permissions After the app (or job) identity exists, grant least-privilege access. Prefer the **`databricks permissions`** CLI where supported for warehouses, catalogs, and serving endpoints (exact resource types and verbs follow current CLI docs). Pattern: ```bash # Illustrative — replace resource names and principal with yours databricks permissions update sql warehouses <warehouse-id> \ --json '{"access_control_list": [{"group_name": "<sp-or-group>", "permission_level": "CAN_USE"}]}' databricks permissions update registered-models <full-model-name> \ --json '{"access_control_list": [{"service_principal_name": "<app-sp-application-id>", "permission_level": "CAN_QUERY"}]}' ``` For UC SQL grants (tables, functions), use `GRANT` in SQL as needed. OTEL or trace tables need explicit `SELECT` / `MODIFY` if your app writes telemetry to UC. --- ## Verify Deployment ```python from databricks.sdk import WorkspaceClient from openai import OpenAI w = WorkspaceClient() client = OpenAI( base_url=f"{w.config.host}/apps/<app-name>/api", api_key=w.config.token, ) response = client.chat.completions.create( model="my-agent", messages=[{"role": "user", "content": "Hello, what can you do?"}], ) print(response.choices[0].message.content) ``` Open the App URL from deploy output in the browser to exercise the hosted UI if applicable. --- ## MCP Integration **Model Context Protocol (MCP)** connects agents to **tools** (Unity Catalog functions, SQL warehouses, retrieval, custom backends) using a standard protocol so the model can invoke capabilities without hard-coding every integration in app code. **Why it matters for deployment:** Tool endpoints and credentials must match the **same identity** the app uses in production (typically the app SP). If MCP reaches a warehouse or UC function, grant that identity the same way you would for in-process tool calls. **Pattern:** 1. Declare MCP **server configs** for each tool class (per Databricks docs: transport, auth, allowed scopes). 2. Register tools with your agent definition so invocations map cleanly to MCP methods. 3. In CI, smoke-test tool calls against a dev workspace before promoting the bundle target. Keep secrets out of source control; use workspace secrets, OIDC, or bundle variables for server URLs and tokens where applicable. See [MCP on Databricks](https://docs.databricks.com/aws/en/generative-ai/mcp/). --- ## Supervisor API: Long-Running Tasks HTTP requests often time out before a **multi-step agent** (plan → tools → synthesis) finishes. The **Supervisor** flow lets you **start** a task, obtain a continuation token or task id, then **poll or resume** with **`task_continue_request`** until the run reaches a terminal state. **When to use:** Long tool chains, human-in-the-loop pauses, or heavy retrieval that cannot complete inside a single synchronous response. **Pattern:** 1. **Start** — initial request returns identifiers needed for continuation (per current API contract). 2. **Continue** — client or backend job sends `task_continue_request` with that context until done or failed. 3. **Persist** — store partial outputs if users disconnect; idempotent continues reduce duplicate side effects where the API allows. Design UIs and APIs to show progress (“still running”) rather than blocking one HTTP call for the full workflow. See [Multi-agent Supervisor: long-running tasks](https://docs.databricks.com/aws/en/generative-ai/agent-bricks/multi-agent-supervisor-long-running-tasks). --- ## Production Trace Linking Reiterate: set **`MLFLOW_ACTIVE_MODEL_ID`** in the app’s **`env`** (see `app.yaml` above) so MLflow GenAI tracing associates production traffic with the **active UC model / app version**. Combine with your Step 5 registration flow so CI or bundle injects the correct ID per deploy. --- ## AI Gateway Integration (Producer Side) If your workspace has an AI Gateway fronting LLM and MCP endpoints (see [foundation/04-ai-gateway](../../foundation/04-ai-gateway/SKILL.md)), configure the deployed agent to **produce** traffic through the gateway rather than hitting provider endpoints directly. This gives the deployed agent: - Uniform usage tracking + inference-table audit per deploy target. - Rate limits that protect the shared provider from a misbehaving release. - Org-wide guardrails (PII, safety) applied before provider calls. - Correlation between MLflow trace `request_id` and gateway inference rows. ### Point the app at the gateway via `app.yaml` env vars ```yaml env: