06-deployment-and-automation

name	06-deployment-and-automation
description	>

Deployment and Automation

Deploy your agent to Databricks Apps (a common production target) and automate evaluate → gate → promote → deploy using Databricks Asset Bundles (DAB) and your CI system.

Upstream Lineage

This skill references Databricks Agent Skills' databricks-mlflow-evaluation skill for evaluate-gate-promote automation, production trace linkage, and monitoring handoff guidance. If release gates depend on eval harness semantics or upstream monitoring patterns, consult the upstream skill first, then apply this skill's Databricks Apps and Asset Bundle deployment contracts.

When to Use

• Evaluation gates passed (SDLC Step 4) and the model is registered in Unity Catalog (Step 5).
• You need a repeatable bundle workflow (databricks bundle deploy / run), not ad-hoc workspace copies.
• You want MCP tool connectivity, Supervisor-style orchestration, or production traces linked to app versions.
• You need service principal access patterns and post-deploy verification.

---

Versioned Resource Path Contract

Every persisted workshop artifact that can be superseded must be versioned. The CI/CD pipeline owned by this skill reads three of these and emits the fourth:

• signoffs (read): /Volumes/<catalog>/<schema>/signoffs/v<N>/decision.md — owned by 04b-stakeholder-signoff
• eval summaries (read): /Volumes/<catalog>/<schema>/eval_runs/v<N>/summary.json — owned by 04-evaluation-runs
• prompt candidates (read): prompts:/{catalog}.{uc_agent_schema}.system_instructions@candidate_v<N> — owned by 01-prompt-registry / 08b-prompt-handauthoring
• deployment plans (emit): /Volumes/<catalog>/<schema>/deployment_plans/v<N>/plan.md

<N> is a monotonically increasing integer per artifact type, scoped to (catalog, schema, artifact_type) (or (catalog, uc_agent_schema, prompt_name) for prompt candidates). Resolve <catalog> and <schema> from state — never hard-code. The promote/deploy step MUST verify that the same <N> is referenced consistently across the eval summary, signoff decision, and prompt candidate before promoting; releasing against a mixed set of versions silently breaks rollback. Always write a fresh v<N+1> for the deployment plan; never overwrite an existing version.

---

Production Registration Gate: Structured Signoff Consumption

The promote step MUST consume the structured YAML front matter in signoffs/v<N>/decision.md (owned by 04b-stakeholder-signoff). Substring grep on the markdown body is forbidden — narrative text legitimately mentions words like "APPROVED" or "REJECTED" in past tense or as quoted reasons.

The signoff document carries two independent decisions. Both must clear before production registration runs:

• engineering_signoff.decision — set to APPROVED or APPROVED_WITH_CONDITIONS.
• stakeholder_signoff.decision — set to APPROVED or APPROVED_WITH_CONDITIONS.

Any of REJECTED, missing block, or unparseable YAML blocks production registration. The single audit-tracked escape hatch is a state_override block in the same front matter that captures the original decision verbatim so reviewers can reconstruct what was bypassed.

# In the CI promote step — runs BEFORE setting the @champion alias
# or calling databricks bundle deploy --target prod.
from pathlib import Path
import yaml

ALLOWED = {"APPROVED", "APPROVED_WITH_CONDITIONS"}

signoff_path = Path(f"/Volumes/{catalog}/{schema}/signoffs") / f"v{version}" / "decision.md"
text = signoff_path.read_text()
assert text.startswith("---\n"), "signoff missing YAML front matter"
_, front, _ = text.split("---\n", 2)
meta = yaml.safe_load(front) or {}

eng = (meta.get("engineering_signoff") or {}).get("decision")
biz = (meta.get("stakeholder_signoff") or {}).get("decision")

if eng not in ALLOWED or biz not in ALLOWED:
    override = meta.get("state_override") or {}
    captured_eng = (override.get("engineering_signoff") or {}).get("decision")
    captured_biz = (override.get("stakeholder_signoff") or {}).get("decision")
    if captured_eng != eng or captured_biz != biz:
        raise SystemExit(
            f"Blocked: engineering={eng}, stakeholder={biz}; no state_override "
            "captures the original decisions. Production registration aborted."
        )
# Only here may the pipeline proceed to alias promotion + bundle deploy.

This gate runs before any of the following actions:

1. Setting the @champion (or production) alias in Unity Catalog.
2. Issuing databricks bundle deploy --target prod.
3. Updating MLFLOW_ACTIVE_MODEL_ID in the production app environment.

If the gate raises, the entire promote step exits non-zero and CI surfaces the failure to the requester. There is no retry without first re-running the signoff workflow in 04b-stakeholder-signoff.

---

Databricks Asset Bundles (DAB)

Genie Code: run every deploy command through runDatabricksCli (pre-authenticated), and be on the bundle's page so the CWD resolves to the bundle root. The CI/CD spine is identical on both clients. See skills/genie-code-environment §3–§4.

Define jobs, apps, and variables in databricks.yml, then deploy and run by target.

# databricks.yml (illustrative)
bundle:
  name: my_agent_bundle

variables:
  catalog: { default: main }
  job_id_env: { default: MY_AGENT_JOB_ID }  # inject app env from bundle if needed

resources:
  jobs:
    my_eval_job:
      name: my-agent-evaluate-and-promote
      # tasks: notebook / wheel / sql — see DAB docs
  apps:
    my_agent_app:
      name: my-agent-app
      source_code_path: ./app

targets:
  dev:
    default: true
    workspace:
      host: https://<dev-workspace>.cloud.databricks.com
  staging:
    workspace:
      host: https://<staging-workspace>.cloud.databricks.com
  prod:
    workspace:
      host: https://<prod-workspace>.cloud.databricks.com

Workflow:

databricks bundle validate
databricks bundle deploy --target dev
databricks bundle run --target dev my_agent_app   # or job name per your bundle

Use targets for environment-specific workspace hosts, variables, and overrides (dev / staging / prod). See Databricks Asset Bundles.

---

Recommended default: Deploy to Databricks Apps for most agents. Use Model Serving only when you need a pure inference endpoint without a custom UI or backend.

Databricks Apps: app.yaml / app.yml

Declare how the app starts and which platform resources it may use.

• command: process that runs your server (for example uvicorn or your framework’s entrypoint).
• env: plain values and bindings. Set MLFLOW_ACTIVE_MODEL_ID to the UC logged-model identifier (or substitute from bundle variables) so production traces align with the deployed app version — see Link production traces to app versions.
• Tracing env vars: also set ENABLE_MLFLOW_TRACING=true, MLFLOW_EXPERIMENT_ID=<numeric>, and APP_ENVIRONMENT=production (or staging) so traces flow from the deployed runtime and app code can override mlflow.source.type via metadata. The full env-var matrix (PAT vs OAuth, SP CAN_EDIT requirement, the Git-folder caveat) lives in the canonical reference: foundation/02-experiment-tracing-and-uc-storage/references/prod-tracing-deployment.md. For the APP_ENVIRONMENT override pattern and user / session metadata, see F2c — Trace context and environments.
• Resources: attach SQL warehouse, serving endpoints, MLflow experiment, Lakebase, etc., per Databricks Apps resources.

Example fragment:

command:
  - "python"
  - "-m"
  - "myapp"

env:
  MLFLOW_ACTIVE_MODEL_ID: "{{ logged_model_id }}"   # resolve via bundle / CI
  # Optional: tracing warehouse, feature flags, etc.

# resources: (warehouse, endpoints, lakebase — per product docs)

Replace {{ logged_model_id }} with your bundle variable or CI-injected value (for example the model version URI or ID your org uses).

---

How app.yaml and databricks.yml Interact

These two files serve different purposes and are read at different times:

File	Read by	When	Purpose
app.yaml	Apps platform	App process startup	Runtime config: command, env vars, resource bindings
databricks.yml	databricks bundle CLI	Deploy time	Provisioning: create/update app, experiments, jobs, permissions

When deploying via bundles, the config block inside databricks.yml (resources.apps.<name>.config) overrides the corresponding fields in app.yaml. Specifically:

• config.command in the bundle replaces command in app.yaml
• config.env in the bundle replaces env in app.yaml
• resources in the bundle app block replaces resources in app.yaml

Recommendation: Use databricks.yml as the source of truth for all deployment configuration. Keep app.yaml as a minimal runtime fallback for local dev or standalone (non-bundle) deploys. Do not maintain the same env vars or resource IDs in both files — they will drift and cause confusing deployment mismatches.

Common mistake: Editing app.yaml to fix a deployed app's config, then wondering why databricks bundle deploy reverts the change. The bundle always writes its own config block.

---

Preflight (Generic)

Before databricks bundle deploy, verify:

• Databricks CLI auth for the intended workspace / target.
• Unity Catalog objects referenced by the app exist (catalogs, schemas, tables, functions).
• The app’s service principal (or run-as identity) has required privileges on warehouses, catalogs, and serving endpoints.
• Attached serving endpoints respond (health / smoke inference if applicable).

Automate these checks in a small script or a DAB job step; do not assume deploy alone validates runtime access.

---

Service Principal Permissions

After the app (or job) identity exists, grant least-privilege access. Prefer the databricks permissions CLI where supported for warehouses, catalogs, and serving endpoints (exact resource types and verbs follow current CLI docs).

Pattern:

# Illustrative — replace resource names and principal with yours
databricks permissions update sql warehouses <warehouse-id> \
  --json '{"access_control_list": [{"group_name": "<sp-or-group>", "permission_level": "CAN_USE"}]}'

databricks permissions update registered-models <full-model-name> \
  --json '{"access_control_list": [{"service_principal_name": "<app-sp-application-id>", "permission_level": "CAN_QUERY"}]}'

For UC SQL grants (tables, functions), use GRANT in SQL as needed. OTEL or trace tables need explicit SELECT / MODIFY if your app writes telemetry to UC.

---

Verify Deployment

from databricks.sdk import WorkspaceClient
from openai import OpenAI

w = WorkspaceClient()
client = OpenAI(
    base_url=f"{w.config.host}/apps/<app-name>/api",
    api_key=w.config.token,
)
response = client.chat.completions.create(
    model="my-agent",
    messages=[{"role": "user", "content": "Hello, what can you do?"}],
)
print(response.choices[0].message.content)

Open the App URL from deploy output in the browser to exercise the hosted UI if applicable.

---

MCP Integration

Model Context Protocol (MCP) connects agents to tools (Unity Catalog functions, SQL warehouses, retrieval, custom backends) using a standard protocol so the model can invoke capabilities without hard-coding every integration in app code.

Why it matters for deployment: Tool endpoints and credentials must match the same identity the app uses in production (typically the app SP). If MCP reaches a warehouse or UC function, grant that identity the same way you would for in-process tool calls.

Pattern:

1. Declare MCP server configs for each tool class (per Databricks docs: transport, auth, allowed scopes).
2. Register tools with your agent definition so invocations map cleanly to MCP methods.
3. In CI, smoke-test tool calls against a dev workspace before promoting the bundle target.

Keep secrets out of source control; use workspace secrets, OIDC, or bundle variables for server URLs and tokens where applicable. See MCP on Databricks.

---

Supervisor API: Long-Running Tasks

HTTP requests often time out before a multi-step agent (plan → tools → synthesis) finishes. The Supervisor flow lets you start a task, obtain a continuation token or task id, then poll or resume with task_continue_request until the run reaches a terminal state.

When to use: Long tool chains, human-in-the-loop pauses, or heavy retrieval that cannot complete inside a single synchronous response.

Pattern:

1. Start — initial request returns identifiers needed for continuation (per current API contract).
2. Continue — client or backend job sends task_continue_request with that context until done or failed.
3. Persist — store partial outputs if users disconnect; idempotent continues reduce duplicate side effects where the API allows.

Design UIs and APIs to show progress (“still running”) rather than blocking one HTTP call for the full workflow. See Multi-agent Supervisor: long-running tasks.

---

Production Trace Linking

Reiterate: set MLFLOW_ACTIVE_MODEL_ID in the app’s env (see app.yaml above) so MLflow GenAI tracing associates production traffic with the active UC model / app version. Combine with your Step 5 registration flow so CI or bundle injects the correct ID per deploy.

---

AI Gateway Integration (Producer Side)

If your workspace has an AI Gateway fronting LLM and MCP endpoints (see foundation/04-ai-gateway), configure the deployed agent to produce traffic through the gateway rather than hitting provider endpoints directly.

This gives the deployed agent:

• Uniform usage tracking + inference-table audit per deploy target.
• Rate limits that protect the shared provider from a misbehaving release.
• Org-wide guardrails (PII, safety) applied before provider calls.
• Correlation between MLflow trace request_id and gateway inference rows.

Point the app at the gateway via app.yaml env vars

env: