03-tools-and-data-access

# Tools and Data Access Give your agent capabilities beyond text generation: query structured data, search documents, invoke custom functions, and connect to external services. This foundational skill covers the Databricks MCP ecosystem that all agent tracks build on. > **Public Preview.** MCP on Databricks is in Public Preview. Refer to the > source documentation links in the References section for the latest server > types, URL patterns, `_meta` parameters, and authentication methods. ## When to Use Use this skill when you are: - Connecting any agent to **Databricks managed MCP servers** (Vector Search, Genie, SQL, UC Functions). - Adding the **code interpreter** (`system.ai.python_exec`) for dynamic Python execution. - Installing **external MCP servers** (GitHub, Glean, Atlassian, or custom). - Connecting to **external services without MCP** via the UC connections proxy, managed OAuth, or UC function `http_request()`. - Using **pre-built tool recipes** (web search, SQL execution, file generation, HTTP API calls) from the tool cookbook — with wiring for all tracks (A/C via `@function_tool`, B via UC Functions). - Adding **runtime guardrails** (input screening, output filtering) to protect against prompt injection, PII leaks, and toxic content. - Configuring **resource grants** for deployed agents (Apps, Model Serving). - Declaring **custom retriever schemas** for evaluation and AI Playground. - Understanding which tool types exist and how they differ before wiring them into a specific track. **Prerequisite:** Foundation Steps 1 and 2 must be complete (MLflow environment, experiment paths, tracing). See [F1](../01-mlflow-genai-foundation/SKILL.md) and [F2](../02-experiment-tracing-and-uc-storage/SKILL.md). --- ## The `databricks-mcp` Library The `databricks-mcp` Python package is the **recommended** way to connect to both managed and external MCP servers on Databricks. It handles authentication, tool discovery, tool invocation, and resource enumeration. ### Install ```bash pip install -U "databricks-mcp" "databricks-sdk" "mcp>=1.9" ``` ### Core Pattern: `DatabricksMCPClient` ```python from databricks.sdk import WorkspaceClient from databricks_mcp import DatabricksMCPClient workspace_client = WorkspaceClient() host = workspace_client.config.host mcp_client = DatabricksMCPClient( server_url=f"{host}/api/2.0/mcp/functions/system/ai", workspace_client=workspace_client, ) tools = mcp_client.list_tools() print(f"Available tools: {[t.name for t in tools]}") result = mcp_client.call_tool("system__ai__python_exec", {"code": "print(42)"}) print(result.content[0].text) ``` ### OAuth Authentication For async code or the standard MCP SDK, use `DatabricksOAuthClientProvider`: ```python from databricks_mcp import DatabricksOAuthClientProvider from mcp.client.streamable_http import streamablehttp_client from mcp.client.session import ClientSession async with streamablehttp_client( url=mcp_server_url, auth=DatabricksOAuthClientProvider(workspace_client), ) as (read_stream, write_stream, _): async with ClientSession(read_stream, write_stream) as session: await session.initialize() tools = await session.list_tools() ``` ### Auto-Discovering Resources for Deployment When logging an agent for Model Serving, `get_databricks_resources()` enumerates every UC object the MCP server exposes — eliminating manual resource listing: ```python from databricks_mcp import DatabricksMCPClient mcp_client = DatabricksMCPClient( server_url=f"{host}/api/2.0/mcp/functions/prod/analytics", workspace_client=workspace_client, ) resources = mcp_client.get_databricks_resources() ``` Pass these resources directly to `mlflow.pyfunc.log_model(resources=...)`. See the Resource Grants section below and [`references/resource-grants.md`](references/resource-grants.md) for the full pattern. --- ## Databricks Managed MCP Servers Databricks provides four managed MCP servers. Unity Catalog permissions are always enforced — agents and users can only access tools and data they are allowed to. | MCP Server | URL Pattern | OAuth Scope | Purpose | |---|---|---|---| | **AI Search** | `/api/2.0/mcp/ai-search/{catalog}/{schema}/{index_name}` | `ai-search` | Semantic search over indexed documents. Index must use Databricks managed embeddings. | | **Genie Space** | `/api/2.0/mcp/genie/{genie_space_id}` | `genie` | Query a single Genie Space to analyze structured data via natural language. Read-only. | | **Genie (cross-space)** | `/api/2.0/mcp/genie` | `genie` | Ask natural-language questions across all Genie Spaces and UC data; returns a grounded answer with a deep link. Read-only. | | **Databricks SQL** | `/api/2.0/mcp/sql` | `sql` | Run AI-generated SQL for data pipelines and ad-hoc queries. Read and write. | | **UC Functions** | `/api/2.0/mcp/functions/{catalog}/{schema}/{function_name}` | `unity-catalog` | Call registered Unity Catalog SQL/Python functions. | > **Naming note:** AI Search was formerly **Vector Search**. The legacy > `/api/2.0/mcp/vector-search/...` URL prefix and `vector-search` OAuth scope > still work for backward compatibility, but `ai-search` is the current > canonical name. To view your MCP servers and their endpoint URLs, go to your workspace > **AI Gateway** > **MCPs**. ### Polling Note The Genie, Genie Space, and Databricks SQL MCP servers run asynchronously and require polling for results of long-running queries (e.g. call `genie_ask`, then poll `genie_poll_response`). Handle this in your tool invocation loop. ### Example: Customer Support Agent ```python host = workspace_client.config.host MANAGED_MCP_SERVER_URLS = [ f"{host}/api/2.0/mcp/ai-search/prod/customer_support/ticket_index", f"{host}/api/2.0/mcp/genie/{billing_space_id}", f"{host}/api/2.0/mcp/functions/prod/billing/lookup_account", ] ``` This gives the agent access to unstructured data (support tickets), structured data (billing tables via Genie), and custom business logic (account lookups). For detailed per-server code examples, see [`references/managed-mcp-servers.md`](references/managed-mcp-servers.md). --- ## `_meta` Parameters The `_meta` parameter is part of the MCP specification. It lets you **preset configuration** for deterministic behavior while keeping queries flexible for the LLM to generate dynamically. ### SQL MCP `_meta` | Parameter | Type | Description | |---|---|---| | `warehouse_id` | str | SQL warehouse ID for executing queries. If not specified, the system selects automatically. | ### Vector Search MCP `_meta` | Parameter | Type | Description | |---|---|---| | `num_results` | int | Number of results to return | | `filters` | str | JSON string of filters (e.g. `'{"updated_after": "2024-01-01"}'`) | | `query_type` | str | `"ANN"` (default) or `"HYBRID"` (vector + keyword) | | `columns` | str | Comma-separated column names to return | | `columns_to_rerank` | str | Comma-separated columns for reranking | | `include_score` | bool | Include similarity score (`"true"` / `"false"`) | | `score_threshold` | float | Minimum similarity score filter | ### Example: Using `_meta` with the MCP SDK ```python from mcp.types import CallToolRequest, CallToolResult request = CallToolRequest( method="tools/call", params={ "name": "execute_sql", "arguments": { "query": "SELECT * FROM my_catalog.my_schema.sales LIMIT 10" }, "_meta": { "warehouse_id": "a1b2c3d4e5f67890" } } ) response = await session.send_request(request, CallToolResult) ``` ### DO / DON'T **DO** — Use `_meta` for configuration that should be deterministic: ```python "_meta": { "warehouse_id": config.get("warehouse_id"), "num_results": "5", "query_type": "HYBRID", } ``` **DON'T** — Put the user's dynamic query in `_meta`: ```python "_meta": {"query": user_question} # WRONG: query is a dynamic argument ``` For complete `_meta` examples per server type, see [`references/managed-mcp-servers.md`](references/managed-mcp-servers.md). ### Dynamic SQL MCP From Agent Tool Plan When SQL MCP is selected through `docs/agent_tool_plan.yaml`, default to read-only schema-scoped access: - Pin `_meta.warehouse_id` from `selected_mcp_servers[].meta.warehouse_id`. - Restrict generated SQL to `SELECT`, `DESCRIBE`, and `EXPLAIN`. - Require fully qualified `catalog.schema.table` references. - Use `selected_mcp_servers[].scope.allowed_tables` when present. - Do not run DDL or DML unless the user explicitly changes `readonly` to false. --- ## External MCP Servers Connect agents to third-party MCP servers through Databricks-managed proxies. The proxy handles authentication and token management; the external server appears as a standard MCP endpoint. ### Four Installation Methods | Method | When to Use | Credential Management | |---|---|---| | **Managed OAuth** | Glean, GitHub, Atlassian (supported providers) | None — Databricks manages OAuth flows | | **Databricks Marketplace** | Curated servers with pre-built integrations | Enter credentials during install | | **Custom HTTP Connection** | Any MCP server (self-hosted or third-party) | Manual — provide bearer token or OAuth creds | | **Dynamic Client Registration** | MCP servers supporting OAuth 2.0 DCR (RFC 7591) | Automatic — DCR handles registration | ### Proxy URL Pattern After installation, every external MCP server is accessible at: ``` https://{workspace_host}/api/2.0/mcp/external/{connection_name} ``` `DatabricksMCPClient` works identically for both managed and external (proxied) servers — add the proxy URL to your server URL list: ```python MANAGED_MCP_SERVER_URLS = [ f"{host}/api/2.0/mcp/functions/system/ai", f"{host}/api/2.0/mcp/external/github_connection", ] ``` ### Per-User vs Shared Principal Authentication | Auth Type | How It Works | Use When | |---|---|---| | **Shared principal** | All users share one set of credentials (bearer token, OAuth M2M, OAuth U2M shared) | External service doesn't need user-specific access | | **Per-user (OAuth U2M Per User)** | Each user authenticates with their own credentials | Accessing user-specific resources (repos, messages, calendars) | ### Testing Before Wiring Test external MCP servers in **AI Playground** without writing code: 1. Go to AI Playground > choose a model with the **Tools enabled** label. 2. Click **Tools > + Add tool > MCP Servers > External MCP servers**. 3. Select your UC connection and chat with the LLM. For detailed installation walkthroughs (including Managed OAuth provider table, Marketplace flow, DCR code, and security guidance), see [`references/external-mcp-connections.md`](references/external-mcp-connections.md). --- ## Code Interpreter (`system.ai.python_exec`) Databricks provides a built-in code interpreter via the `system.ai.python_exec` Unity Catalog function. It lets agents dynamically write and execute Python code — useful for calculations, data transformations, chart generation, or any task better solved with code than natural language. ### Wiring The code interpreter is available as a managed MCP tool at the UC Functions endpoint for `system.ai`: ```python from agents import Agent, Runner from databricks.sdk import WorkspaceClient from databricks_openai.agents import McpServer workspace_client = WorkspaceClient() host = workspace_client.config.host async with McpServer.from_uc_function( catalog="system", schema="ai", function_name="python_exec", workspace_client=workspace_client, name="code-interpreter", ) as code_interpreter: agent = Agent( name="coding-agent", instructions="Use the python_exec tool to run code when calculations or data manipulation are needed.", model="databricks-claude-sonnet-4-6", mcp_servers=[code_interpreter], ) result = await Runner.run(agent, "Calculate the first 10 Fibonacci numbers") print(result.final_output) ``` Alternatively, use the `DatabricksMCPClient` pattern: ```python mcp_client = DatabricksMCPClient( server_url=f"{host}/api/2.0/mcp/functions/system/ai/python_exec", workspace_client=workspace_client, ) result = mcp_client.call_tool("system__ai__python_exec", {"code": "print(sum(range(100)))"}) ``` ### Use Cases - **Calculations:** Agent needs to compute financial metrics, statistics, or math that shouldn't be approximated by the LLM. - **Data transformation:** Parse CSV/JSON, reshape data, apply business rules. - **Chart generation:** Create matplotlib/plotly visualizations from query results. - **Validation:** Run deterministic checks on data before returning answers. ### Safety Notes - The code interpreter runs in a **sandboxed environment** — it cannot access the local filesystem, network, or workspace resources beyond what the function's identity allows. - Execution is **stateless** — each `python_exec` call starts fresh with no shared memory between invocations. - Use clear instructions to tell the agent **when** to use code execution vs. other tools (e.g., "Use python_exec for calculations, use SQL MCP for data queries"). ### Resource Grant Grant the app's service principal access to the function in `databricks.yml`: ```yaml resources: