03-tools-and-data-access

name	03-tools-and-data-access
description	Use when connecting any agent to data and external capabilities. Covers Databricks managed MCP servers (Vector Search, Genie, SQL, UC Functions), the databricks-mcp client library, _meta parameters, external MCP servers, code interpreter, non-MCP connections, resource grants, custom retriever schemas, tool cookbook (web search, SQL, file generation, HTTP), runtime guardrails, and end-to-end external MCP examples. Foundation Step 3. Consumed by all agent tracks (A, B, C).
license	Apache-2.0
clients	["ide_cli","genie_code"]
bundle_resource	none
deploy_verb	none
deploy_note	Agent tools + managed/external MCP servers + UC resource grants — code + grants, no bundle resource. Tooling resolves identically on both clients; on Genie Code use its built-in tool surface and run grant CLI steps through runDatabricksCli. See `skills/genie-code-environment`.
coverage	full
metadata	{"last_verified":"2026-06-05","volatility":"high","upstream_sources":[],"author":"prashanth-subrahmanyam","version":"1.2.0","domain":"genai-agents","pipeline_position":"F3","consumes":"mlflow_environment, experiment_paths","produces":"mcp_server_knowledge, resource_grants, retriever_schemas, databricks_mcp_client","grounded_in":"docs.databricks.com/aws/en/generative-ai/mcp/managed-mcp, docs.databricks.com/aws/en/generative-ai/mcp/external-mcp, docs.databricks.com/aws/en/generative-ai/agent-framework/build-agent-tool, docs.databricks.com/aws/en/generative-ai/agent-framework/mcp-server"}
fields_read	["resources.knowledge_base_documents","resources.genie_spaces","resources.vector_search_indexes","agent.tools","agent.mcp_servers","agent.knowledge_base_backend","agent.external_integrations"]

Tools and Data Access

Give your agent capabilities beyond text generation: query structured data, search documents, invoke custom functions, and connect to external services. This foundational skill covers the Databricks MCP ecosystem that all agent tracks build on.

Public Preview. MCP on Databricks is in Public Preview. Refer to the source documentation links in the References section for the latest server types, URL patterns, _meta parameters, and authentication methods.

When to Use

Use this skill when you are:

• Connecting any agent to Databricks managed MCP servers (Vector Search, Genie, SQL, UC Functions).
• Adding the code interpreter (system.ai.python_exec) for dynamic Python execution.
• Installing external MCP servers (GitHub, Glean, Atlassian, or custom).
• Connecting to external services without MCP via the UC connections proxy, managed OAuth, or UC function http_request().
• Using pre-built tool recipes (web search, SQL execution, file generation, HTTP API calls) from the tool cookbook — with wiring for all tracks (A/C via @function_tool, B via UC Functions).
• Adding runtime guardrails (input screening, output filtering) to protect against prompt injection, PII leaks, and toxic content.
• Configuring resource grants for deployed agents (Apps, Model Serving).
• Declaring custom retriever schemas for evaluation and AI Playground.
• Understanding which tool types exist and how they differ before wiring them into a specific track.

Prerequisite: Foundation Steps 1 and 2 must be complete (MLflow environment, experiment paths, tracing). See F1 and F2.

---

The databricks-mcp Library

The databricks-mcp Python package is the recommended way to connect to both managed and external MCP servers on Databricks. It handles authentication, tool discovery, tool invocation, and resource enumeration.

Install

pip install -U "databricks-mcp" "databricks-sdk" "mcp>=1.9"

Core Pattern: DatabricksMCPClient

from databricks.sdk import WorkspaceClient
from databricks_mcp import DatabricksMCPClient

workspace_client = WorkspaceClient()
host = workspace_client.config.host

mcp_client = DatabricksMCPClient(
    server_url=f"{host}/api/2.0/mcp/functions/system/ai",
    workspace_client=workspace_client,
)

tools = mcp_client.list_tools()
print(f"Available tools: {[t.name for t in tools]}")

result = mcp_client.call_tool("system__ai__python_exec", {"code": "print(42)"})
print(result.content[0].text)

OAuth Authentication

For async code or the standard MCP SDK, use DatabricksOAuthClientProvider:

from databricks_mcp import DatabricksOAuthClientProvider
from mcp.client.streamable_http import streamablehttp_client
from mcp.client.session import ClientSession

async with streamablehttp_client(
    url=mcp_server_url,
    auth=DatabricksOAuthClientProvider(workspace_client),
) as (read_stream, write_stream, _):
    async with ClientSession(read_stream, write_stream) as session:
        await session.initialize()
        tools = await session.list_tools()

Auto-Discovering Resources for Deployment

When logging an agent for Model Serving, get_databricks_resources() enumerates every UC object the MCP server exposes — eliminating manual resource listing:

from databricks_mcp import DatabricksMCPClient

mcp_client = DatabricksMCPClient(
    server_url=f"{host}/api/2.0/mcp/functions/prod/analytics",
    workspace_client=workspace_client,
)
resources = mcp_client.get_databricks_resources()

Pass these resources directly to mlflow.pyfunc.log_model(resources=...). See the Resource Grants section below and references/resource-grants.md for the full pattern.

---

Databricks Managed MCP Servers

Databricks provides four managed MCP servers. Unity Catalog permissions are always enforced — agents and users can only access tools and data they are allowed to.

MCP Server	URL Pattern	OAuth Scope	Purpose
AI Search	/api/2.0/mcp/ai-search/{catalog}/{schema}/{index_name}	ai-search	Semantic search over indexed documents. Index must use Databricks managed embeddings.
Genie Space	/api/2.0/mcp/genie/{genie_space_id}	genie	Query a single Genie Space to analyze structured data via natural language. Read-only.
Genie (cross-space)	/api/2.0/mcp/genie	genie	Ask natural-language questions across all Genie Spaces and UC data; returns a grounded answer with a deep link. Read-only.
Databricks SQL	/api/2.0/mcp/sql	sql	Run AI-generated SQL for data pipelines and ad-hoc queries. Read and write.
UC Functions	/api/2.0/mcp/functions/{catalog}/{schema}/{function_name}	unity-catalog	Call registered Unity Catalog SQL/Python functions.

Naming note: AI Search was formerly Vector Search. The legacy /api/2.0/mcp/vector-search/... URL prefix and vector-search OAuth scope still work for backward compatibility, but ai-search is the current canonical name.

To view your MCP servers and their endpoint URLs, go to your workspace > AI Gateway > MCPs.

Polling Note

The Genie, Genie Space, and Databricks SQL MCP servers run asynchronously and require polling for results of long-running queries (e.g. call genie_ask, then poll genie_poll_response). Handle this in your tool invocation loop.

Example: Customer Support Agent

host = workspace_client.config.host

MANAGED_MCP_SERVER_URLS = [
    f"{host}/api/2.0/mcp/ai-search/prod/customer_support/ticket_index",
    f"{host}/api/2.0/mcp/genie/{billing_space_id}",
    f"{host}/api/2.0/mcp/functions/prod/billing/lookup_account",
]

This gives the agent access to unstructured data (support tickets), structured data (billing tables via Genie), and custom business logic (account lookups).

For detailed per-server code examples, see references/managed-mcp-servers.md.

---

_meta Parameters

The _meta parameter is part of the MCP specification. It lets you preset configuration for deterministic behavior while keeping queries flexible for the LLM to generate dynamically.

SQL MCP _meta

Parameter	Type	Description
warehouse_id	str	SQL warehouse ID for executing queries. If not specified, the system selects automatically.

Vector Search MCP _meta

Parameter	Type	Description
num_results	int	Number of results to return
filters	str	JSON string of filters (e.g. '{"updated_after": "2024-01-01"}')
query_type	str	"ANN" (default) or "HYBRID" (vector + keyword)
columns	str	Comma-separated column names to return
columns_to_rerank	str	Comma-separated columns for reranking
include_score	bool	Include similarity score ("true" / "false")
score_threshold	float	Minimum similarity score filter

Example: Using _meta with the MCP SDK

from mcp.types import CallToolRequest, CallToolResult

request = CallToolRequest(
    method="tools/call",
    params={
        "name": "execute_sql",
        "arguments": {
            "query": "SELECT * FROM my_catalog.my_schema.sales LIMIT 10"
        },
        "_meta": {
            "warehouse_id": "a1b2c3d4e5f67890"
        }
    }
)

response = await session.send_request(request, CallToolResult)

DO / DON'T

DO — Use _meta for configuration that should be deterministic:

"_meta": {
    "warehouse_id": config.get("warehouse_id"),
    "num_results": "5",
    "query_type": "HYBRID",
}

DON'T — Put the user's dynamic query in _meta:

"_meta": {"query": user_question}  # WRONG: query is a dynamic argument

For complete _meta examples per server type, see references/managed-mcp-servers.md.

Dynamic SQL MCP From Agent Tool Plan

When SQL MCP is selected through docs/agent_tool_plan.yaml, default to read-only schema-scoped access:

• Pin _meta.warehouse_id from selected_mcp_servers[].meta.warehouse_id.
• Restrict generated SQL to SELECT, DESCRIBE, and EXPLAIN.
• Require fully qualified catalog.schema.table references.
• Use selected_mcp_servers[].scope.allowed_tables when present.
• Do not run DDL or DML unless the user explicitly changes readonly to false.

---

External MCP Servers

Connect agents to third-party MCP servers through Databricks-managed proxies. The proxy handles authentication and token management; the external server appears as a standard MCP endpoint.

Four Installation Methods

Method	When to Use	Credential Management
Managed OAuth	Glean, GitHub, Atlassian (supported providers)	None — Databricks manages OAuth flows
Databricks Marketplace	Curated servers with pre-built integrations	Enter credentials during install
Custom HTTP Connection	Any MCP server (self-hosted or third-party)	Manual — provide bearer token or OAuth creds
Dynamic Client Registration	MCP servers supporting OAuth 2.0 DCR (RFC 7591)	Automatic — DCR handles registration

Proxy URL Pattern

After installation, every external MCP server is accessible at:

https://{workspace_host}/api/2.0/mcp/external/{connection_name}

DatabricksMCPClient works identically for both managed and external (proxied) servers — add the proxy URL to your server URL list:

MANAGED_MCP_SERVER_URLS = [
    f"{host}/api/2.0/mcp/functions/system/ai",
    f"{host}/api/2.0/mcp/external/github_connection",
]

Per-User vs Shared Principal Authentication

Auth Type	How It Works	Use When
Shared principal	All users share one set of credentials (bearer token, OAuth M2M, OAuth U2M shared)	External service doesn't need user-specific access
Per-user (OAuth U2M Per User)	Each user authenticates with their own credentials	Accessing user-specific resources (repos, messages, calendars)

Testing Before Wiring

Test external MCP servers in AI Playground without writing code:

1. Go to AI Playground > choose a model with the Tools enabled label.
2. Click Tools > + Add tool > MCP Servers > External MCP servers.
3. Select your UC connection and chat with the LLM.

For detailed installation walkthroughs (including Managed OAuth provider table, Marketplace flow, DCR code, and security guidance), see references/external-mcp-connections.md.

---

Code Interpreter (system.ai.python_exec)

Databricks provides a built-in code interpreter via the system.ai.python_exec Unity Catalog function. It lets agents dynamically write and execute Python code — useful for calculations, data transformations, chart generation, or any task better solved with code than natural language.

Wiring

The code interpreter is available as a managed MCP tool at the UC Functions endpoint for system.ai:

from agents import Agent, Runner
from databricks.sdk import WorkspaceClient
from databricks_openai.agents import McpServer

workspace_client = WorkspaceClient()
host = workspace_client.config.host

async with McpServer.from_uc_function(
    catalog="system",
    schema="ai",
    function_name="python_exec",
    workspace_client=workspace_client,
    name="code-interpreter",
) as code_interpreter:
    agent = Agent(
        name="coding-agent",
        instructions="Use the python_exec tool to run code when calculations or data manipulation are needed.",
        model="databricks-claude-sonnet-4-6",
        mcp_servers=[code_interpreter],
    )
    result = await Runner.run(agent, "Calculate the first 10 Fibonacci numbers")
    print(result.final_output)

Alternatively, use the DatabricksMCPClient pattern:

mcp_client = DatabricksMCPClient(
    server_url=f"{host}/api/2.0/mcp/functions/system/ai/python_exec",
    workspace_client=workspace_client,
)
result = mcp_client.call_tool("system__ai__python_exec", {"code": "print(sum(range(100)))"})

Use Cases

• Calculations: Agent needs to compute financial metrics, statistics, or math that shouldn't be approximated by the LLM.
• Data transformation: Parse CSV/JSON, reshape data, apply business rules.
• Chart generation: Create matplotlib/plotly visualizations from query results.
• Validation: Run deterministic checks on data before returning answers.

Safety Notes

• The code interpreter runs in a sandboxed environment — it cannot access the local filesystem, network, or workspace resources beyond what the function's identity allows.
• Execution is stateless — each python_exec call starts fresh with no shared memory between invocations.
• Use clear instructions to tell the agent when to use code execution vs. other tools (e.g., "Use python_exec for calculations, use SQL MCP for data queries").

Resource Grant

Grant the app's service principal access to the function in databricks.yml:

resources: