04-authentication

# Agent authentication (Databricks) Doc-driven skill for how AI agents authenticate on **Databricks Apps** and **Model Serving**. Pair with Track A Step 2 (`agent_class`) when wiring an agent implementation to the right client and grants. Primary sources: - [Authentication for AI agents (Databricks Apps)](https://docs.databricks.com/aws/en/generative-ai/agent-framework/agent-authentication) - [Authentication for AI agents (Model Serving)](https://docs.databricks.com/aws/en/generative-ai/agent-framework/agent-authentication-model-serving) ## When to Use Use this skill whenever you are: - Configuring **permissions** for a deployed agent (SP vs user, scopes, resources). - Implementing **user authorization (OBO)** in agent handlers. - Declaring **MLflow `resources`** for **automatic authentication passthrough** on Model Serving. - Supporting **dual deployment** (Apps + Model Serving + local) with one codebase. - Connecting to **Databricks MCP servers** or **external MCP** from an agent. **Cross-step impact:** The auth patterns you establish here affect both Track A evaluation and deployment (A6/A7) and the SDLC pipeline. Environment detection (local vs Apps vs Model Serving) must work in all three contexts — especially during [A6: Evaluation](../06-evaluation/SKILL.md) (which runs under your CLI credentials, not OBO) and [S6: Deployment & Automation](../../../sdlc/06-deployment-and-automation/SKILL.md) (which provisions a system SP for production). ## Two authentication methods overview | Method | Description | When to use | | --- | --- | --- | | **App authorization** | Agent uses an **auto-created service principal** with consistent permissions (formerly “SP authentication”). | **Most common.** All users share the same access; shared indexes, shared experiments, uniform behavior. | | **User authorization** | Agent acts as the **requesting user** (OBO): downscoped token, per-request context. | **Per-user** permissions, **audit trails**, Unity Catalog row filters/column masks, fine-grained UC access. | You can **combine** both in one agent (for example, app SP for a shared Vector Search index and user client for user-specific tables). ## App authorization (default for Databricks Apps) - Databricks **automatically creates a service principal** when the app is created; that SP is the app identity. - **All users** of the app share the **same** permissions granted to that SP. - Grant the SP access via the **Databricks Apps UI** (Edit app → Configure → resources) or declaratively in **`databricks.yml`** / bundle app resources. See [Apps in bundles](https://docs.databricks.com/aws/en/dev-tools/bundles/resources#app). ### MLflow experiment The agent needs an MLflow experiment for traces and evaluation. Grant the SP **Can Edit** on that experiment and add it as an app resource. See [Add an MLflow experiment resource](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/mlflow). ### Minimum permissions (common resources) Align grants with official tables; typical **minimum** expectations for agents: | Resource | Permission | | --- | --- | | MLflow experiment | `Can Edit` | | SQL warehouse | `Can Use` | | Model Serving endpoint | `Can Query` | | Genie space | `Can Run` | | Vector Search index | `Can Select` | | Unity Catalog table | `SELECT` (plus `USE SCHEMA` / `USE CATALOG` on parents) | | Unity Catalog connection | `Use Connection` | | Unity Catalog volume | `Can Read` or `Can Read and Write` | | Lakebase | `Can Connect and Create` | For the complete permission matrix (all resource types, grant methods, dependency chains), see: `references/resource-permissions.md`. > **Bundle-kind selection:** In `databricks.yml`, warehouses, serving endpoints, and UC connections have their own top-level kinds (`sql_warehouse`, `serving_endpoint`, `connection`). UC functions/tables/schemas/models use the **`uc_securable`** kind with the right `securable_type`. Another Databricks App (e.g. when an AppKit frontend proxies to an Agent App backend) uses the **`app`** kind with `CAN_USE`. See [`03-tools-and-mcp`](../03-tools-and-mcp/SKILL.md#resource-kind-quick-reference) for the full kind-selection table and the `databricks bundle schema` verification command. For **Model Serving** automatic passthrough, permission names and **minimum MLflow versions** per resource type differ; see the [Model Serving authentication](https://docs.databricks.com/aws/en/generative-ai/agent-framework/agent-authentication-model-serving) page. **Prompt Registry (UC):** grant **`CREATE FUNCTION`**, **`EXECUTE`**, and **`MANAGE`** on the UC **schema** used to store prompts. **Downstream dependencies:** If you grant access to a composite resource (for example a **Genie space**), you must also grant access to **dependent** objects (underlying tables, warehouses, UC functions, etc.). Missing downstream grants surface as **permission denied at runtime**. **Who can attach resources:** The user adding a resource needs **Can Manage** on **both** the resource and the app (per product docs). Security hardening: follow [App authorization](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#app-authorization) for least privilege and credential handling. For complete app authorization patterns (SP permission granting via UI/CLI/SDK, `databricks.yml` declarations, least privilege patterns, audit and rotation), see: `references/app-authorization.md`. ## User authorization (OBO) for Databricks Apps User authorization is **Public Preview** as of product documentation; workspace admins may need to enable it. ### How it works 1. **Add API scopes** in the app so Databricks knows which APIs the app may call on behalf of users. See [Add scopes to an app](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#add-scopes-to-an-app). 2. Databricks **downscopes** the user's credentials to those scopes. 3. The downscoped token is forwarded via the **`x-forwarded-access-token`** HTTP header. 4. **MLflow Agent Server** stores the token **per request** for use in agent code. Resources reachable with user authorization (per docs) include SQL warehouses, Genie spaces, files/directories, Model Serving endpoints, Vector Search indexes, UC connections, and UC tables. ### Canonical Databricks Apps user headers Databricks Apps sets the following canonical headers on every inbound request. Read these directly — there is no JSON-blob "user info" header. - `x-forwarded-email` — the end user's email (canonical attribute used as MLflow `AssessmentSource.source_id`). - `x-forwarded-preferred-username` — the user's preferred username (often equals the email). - `x-forwarded-user` — the user's stable workspace user id. - `x-forwarded-access-token` — the downscoped OBO token for user-scoped Databricks API calls (only present when the App declares `user_api_scopes`). `x-forwarded-user-info` is **not** a canonical Databricks Apps header and must not be used. Earlier internal patterns assumed it existed; it does not. ### Pathway-C: outbound `x-app-user-email` from AppKit to the Agent App When the AppKit App (frontend) is a separate Databricks App from the Agent App (backend) — the 2-Apps / Variant 4 / Pathway-C deployment — the AppKit proxy must set `x-app-user-email` on outbound requests to the Agent App. The agent uses this value as `AssessmentSource.source_id` for MLflow feedback (see [SDLC 04c-end-user-feedback](../../../sdlc/04c-end-user-feedback/SKILL.md)). **Do not infer originating user identity from the app-to-app OAuth token.** The `Authorization: Bearer` header on the inbound agent request represents the **AppKit App service principal hop** between the two Databricks Apps, not the originating end user. If the agent calls `WorkspaceClient().current_user.me()` on that Bearer, it gets the AppKit SP — not the user — and every assessment will be attributed to the same SP UUID. Concretely, in the Agent App's `@invoke` / `@stream` handler: ```python from mlflow.genai import agent_server import mlflow @agent_server.invoke async def handle_invoke(request: dict, http_request) -> dict: user_email = http_request.headers.get("x-app-user-email") # set by AppKit proxy # ... run the agent ... mlflow.log_feedback( trace_id=trace_id, name="user_feedback", value=is_upvoted, source=mlflow.entities.AssessmentSource( source_type="HUMAN", source_id=user_email, # NOT the SP from the Bearer ), ) ``` The proxy-side wire-up (TypeScript) lives in [`apps_lakebase/skills/06d-appkit-agent-app-proxy/SKILL.md`](../../../../apps_lakebase/skills/06d-appkit-agent-app-proxy/SKILL.md) under "Header Contract". ### Declare API scopes in `app.yaml` (canonical) Declare the scopes your agent needs in the App's `app.yaml` under `user_api_scopes`. The set should be **least privilege** — list only the APIs the agent actually calls on behalf of the user. ```yaml # app.yaml — Agent App command: - mlflow - genai - agent-server - serve - --module - agent user_api_scopes: - serving.serving-endpoints # call LLM + KA endpoints as the user - dashboards.genie # call Genie spaces as the user - sql.statement-execution # run SQL with user warehouse permissions - vectorsearch.vector-search-endpoints # if using Vector Search - catalog.connections # if calling external MCP via UC connection ``` When a Databricks App declares scopes, the user must consent the first time they open the app. Token downscoping is enforced by Databricks — your code just reads `x-forwarded-access-token` from the incoming request. ### Implement in code (Option B canonical) Import the helper that matches your wrapping pattern: ```python # Option B (canonical) — module-level @invoke / @stream on Apps: from databricks_app.utils import get_user_workspace_client # Option A (legacy) — class-based ResponsesAgent on Model Serving: # from agent_server.utils import get_user_workspace_client ``` **CRITICAL:** Call **`get_user_workspace_client()` inside the request handler** (`@invoke` / `@stream` for Option B, or `predict` / `predict_stream` for Option A), **not** in `__init__` or app startup. There is **no user context** at startup; initializing early **fails** or uses the wrong identity. ```python # Option B canonical — receive the http_request, hand it to the helper from mlflow.genai import agent_server from databricks_app.utils import get_user_workspace_client @agent_server.invoke async def handle_invoke(request: dict, http_request) -> dict: ws = get_user_workspace_client(http_request) # user-scoped response = ws.serving_endpoints.query( name="my-endpoint", messages=request["input"], ) return {"output": response.choices[0].message.content} ``` For scope design and escalation considerations, see [Scope-based security and privilege escalation](https://docs.databricks.com/aws/en/dev-tools/databricks-apps/auth#scope-based). For complete OBO implementation patterns (full agent class, scope walkthrough, token lifecycle, testing locally vs deployed, combining app + user auth), see: `references/user-authorization.md`. ## Automatic auth passthrough (Model Serving) For agents **logged and served** on Model Serving, **declare dependencies** at **`mlflow.pyfunc.log_model`** time. Databricks **provisions and rotates short-lived credentials** (M2M OAuth-style) for a **system SP** tied to the model version so the agent can call declared resources—**no manual PAT wiring** for those resources. Notes from product docs: - Endpoint **creator** must be able to access all logged resources. - The system SP for the model version may **not** appear in normal SP listings; it is removed if the model version is removed from the endpoint. - Log **all downstream** dependencies (same principle as Apps). Example pattern (trim `resources` to what your agent actually uses): ```python from mlflow.models.resources import ( DatabricksServingEndpoint, DatabricksVectorSearchIndex, DatabricksLakebase, ) mlflow.pyfunc.log_model( python_model="/path/to/agent.py", resources=[ DatabricksServingEndpoint(endpoint_name="my-llm"), DatabricksVectorSearchIndex(index_name="prod.docs.index"), DatabricksLakebase(database_instance_name="my-lakebase"), ], ) ``` See [log agent](https://docs.databricks.com/aws/en/generative-ai/agent-framework/log-agent) and [mlflow.pyfunc.log_model](https://mlflow.org/docs/latest/api_reference/python_api/mlflow.pyfunc.html#mlflow.pyfunc.log_model) for full `resources` types, `AuthPolicy` / OBO on Model Serving, and MLflow version minimums per resource. For complete Model Serving auth patterns (all resource types, AuthPolicy, system SP lifecycle, version requirements), see: `references/model-serving-auth-passthrough.md`. ## Environment detection for dual deployment Use environment signals to pick **passthrough resources** vs **Apps OBO** vs **local development**: ```python import os if os.environ.get("IS_IN_DB_MODEL_SERVING_ENV"): # Model Serving: rely on resources declared at log_model time (and optional OBO policies) pass elif os.environ.get("DATABRICKS_APP_NAME"): # Databricks Apps: user auth via Agent Server helper when using OBO from agent_server.utils import get_user_workspace_client client = get_user_workspace_client() else: # Local / notebook: default workspace client (user or configured auth) from databricks.sdk import WorkspaceClient client = WorkspaceClient() ``` Treat this as a **pattern**; exact env vars and client construction should match your template and MLflow/Databricks versions. For complete OBO authentication patterns (environment detection, error handling, token refresh, production learnings), see: `references/obo-authentication.md`. ## MCP server authentication ### Databricks managed MCP - Declare **all required resources** in **`databricks.yml`** / bundle and in **MLflow `resources`** when applicable. - Grant the app's **service principal** (app authorization) or **users** (user authorization) access to **every downstream** UC object the MCP paths touch (for example vector indexes and UC functions under the schemas used in MCP URLs). - Product docs illustrate URL patterns under `/api/2.0/mcp/...`; ensure grants cover **each** index/schema/function namespace your agent uses. - **Auth header source.** When constructing `MCPServerSse` inside an `@invoke` / `@stream` handler, pull the bearer token from the user-scoped client to preserve OBO end-to-end: ```python ws = get_user_workspace_client(http_request) ai_search_mcp = MCPServerSse( # AI Search (formerly Vector Search); legacy /mcp/vector-search/ prefix still works url=f"https://{ws.config.host}/api/2.0/mcp/ai-search/prod/docs/idx", headers={"Authorization": f"Bearer {ws.config.token}"}, ) ``` Building MCP clients at module load uses the SP token instead — fine for app-authorization, wrong for user-authorization. For managed MCP, consider **`databricks_mcp.DatabricksMCPClient().get_databricks_resources()`** (from the `databricks-mcp` package) to enumerate dependencies—see [Authentication for AI agents (Model Serving)](https://docs.databricks.com/aws/en/generative-ai/agent-framework/agent-authentication-model-serving). ### External MCP servers