databricks-solutions

Lakebase Change Data Feed (CDF) — stream INSERT/UPDATE/DELETE from Lakebase Postgres via the WAL (wal2delta) into Unity Catalog lb_<table>_history Delta tables. Use for the sync_from_lakebase step when enable_lakebase_cdf is true: prerequisites (Postgres 17…

Standard patterns for Databricks Asset Bundles configuration files for serverless jobs, DLT pipelines, dashboards, alerts, apps, and workflows. Use when creating, configuring, or deploying DABs for infrastructure-as-code deployments. Covers mandatory…

Session-start behavioral manifest for running this workshop inside Databricks Genie Code. Read FIRST when the client is Genie Code (the coding agent embedded in the Databricks workspace) so you begin knowing how it behaves — surface/page tool-scoping, the…

End-to-end orchestrator for designing complete Gold layer schemas with ERDs, YAML files, lineage tracking, and comprehensive business documentation. Guides users through dimensional modeling, ERD creation (master/domain/summary based on table count), YAML…

Cross-validation of Gold layer design artifacts during the design phase. Use when validating that YAML schemas, ERDs, lineage CSVs, and PK/FK references are internally consistent before handing off to implementation. Catches design-time inconsistencies (e.g.,…

Owns the end-to-end state, gating, pathway-applicability, and retrospective contract for the `Instructions.md` / `example/<use_case>/WALKTHROUGH.md` vibecoding workshops. Every workshop prompt invokes this skill at the top (`enter`) and bottom (`exit`) so the…

End-to-end autonomous deployment and operations skill. Deploys Databricks Asset Bundles, runs jobs and pipelines, polls for completion, diagnoses failures, applies fixes, redeploys, and verifies — all without human intervention. Also serves as SDK/CLI/REST…

Use when any agent or downstream skill needs Unity Catalog schemas and managed volumes (knowledge_sources, agent_outputs, memory tables, benchmark tables, etc.) provisioned idempotently. Foundation Step 0 — runs before MLflow tracing setup and before any…

Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents…

Add memory capabilities to your agent. Use when: (1) User asks about 'memory', 'state', 'remember', 'conversation history', (2) Want to persist conversations or user preferences, (3) Adding checkpointing or long-term storage.

Analyzes a single MLflow trace to answer a user query about it. Use when the user provides a trace ID and asks to debug, investigate, find issues, root-cause errors, understand behavior, or analyze quality. Triggers on "analyze this trace", "what went wrong…

Build apps on Databricks Apps platform. Use when asked to create dashboards, data apps, analytics tools, or visualizations. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.

Databricks CLI operations: auth, profiles, data exploration, and bundles. Contains up-to-date guidelines for Databricks-related CLI tasks.

Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.

Databricks Lakebase Postgres: projects, scaling, connectivity, Lakebase synced tables, and Data API. Use when asked about Lakebase databases, OLTP storage, or connecting apps to Postgres on Databricks.

Manage Databricks Model Serving endpoints via CLI. Use when asked to create, configure, query, or manage model serving endpoints for LLM inference, custom models, or external models.

Build PWAF-compliant ISV integrations with Databricks: OAuth, telemetry (User-Agent), Unity Catalog, JDBC, SDK, SQL drivers, REST API, Databricks Connect.

PWAF-compliant Python SQL Connector (databricks-sql-connector): PAT, OAuth M2M, OAuth U2M (custom OAuth app PKCE + token-env), credentials_provider patterns, error handling, retry logic. Use when building Python integrations that run SQL queries via a…

Add a Databricks connector to an existing project that has no Databricks integration. Use when your product already exists and you want to add Databricks as a new data source or backend.

How to write a build_report.md for any PWAF connector. Captures skill traceability, sufficiency assessment, and test error/fix log. Use after implementing any connector.

How to structure a Databricks connector (REST or Python SDK): config, connect, operations, validation. Use when designing or building a new connector.

How to build and use a PWAF connector test runner (tests/run_all_tests.sh). Covers env isolation, auth types, single connector or single auth mode, parallel execution, browser tests, and report generation.

PWAF-compliant Databricks Connect (Python): PAT, OAuth M2M, OAuth U2M; serverless and classic compute. Use when building or testing Spark-over-Connect integrations.

PWAF-compliant Databricks SDK for Go (databricks-sdk-go): PAT, OAuth M2M, U2M token-env, U2M custom OAuth app (PKCE); useragent.WithProduct/WithPartner. Use when building or testing Go SDK workspace API integrations.

Build Databricks Apps from this cookbook: minimum folder layout and app.yaml per framework (Dash, Streamlit, Reflex, FastAPI), multi-page wiring, docs category-first workflow then README recipes, runtime rules, combining recipes. Feature code only from repo…

Follow-up to build-app: wrap the chosen cookbook framework (Dash, Streamlit, Reflex, FastAPI) in a Databricks Asset Bundle (DAB)—framework-specific src/app layout, app.yaml commands, bundle validate/deploy/run, CI/CD secrets. Requires a working app from…

Invoke ML models, run vector search, and connect to MCP servers from Databricks Apps. Use when the user wants to call a model serving endpoint, query a vector search index, use LLMs/embeddings, or connect to an MCP server in Dash, Streamlit, Reflex, or…

Implement authentication in Databricks Apps: get current user info, on-behalf-of-user (OBO) auth, service principal auth, and OAuth patterns. Use when the user asks about auth, user tokens, service principals, OAuth, or OBO in Dash, Streamlit, Reflex, or…

Embed AI/BI dashboards and integrate Genie Spaces into Databricks Apps. Use when the user wants to embed a Databricks dashboard or build a chat interface with the Genie Conversations API in Dash, Streamlit, or Reflex.

Connect Databricks Apps to shared clusters or serverless compute using Databricks Connect. Use when the user needs to run Spark SQL or Python on a cluster from a Dash, Streamlit, or Reflex app.

Connect Databricks Apps to external services using Unity Catalog HTTP connections and retrieve Databricks secrets. Use when the user needs to call external APIs (GitHub, Jira, Slack), set up MCP connections, or access secrets from Dash, Streamlit, or Reflex…

Read and write Delta tables and connect to Lakebase (OLTP PostgreSQL) from Databricks Apps. Use when the user needs to query UC tables, edit table data, or use Lakebase for transactional workloads in Dash, Streamlit, Reflex, or FastAPI.

Guide an agentops-stacks project through its full production lifecycle — data preparation, agent development, evaluation gates, CI/CD promotion, and production monitoring — following the Single-Account Single-Agent pattern from the Big Book of AgentOps. Use…

Operate and troubleshoot the UC Functions tools component in an AgentOps Stacks project — register new functions, update EXECUTE grants, list available functions, re-run the registration job. Use when the user needs to add, update, or grant access to UC…

Set up and operate usage tracking / cost attribution for an AgentOps Stacks project — tag resources with project/environment/agent, attach a serverless usage policy, retrofit tags onto a project scaffolded without them, tag newly added compute, and query…

Add a new agent to an existing AgentOps Stacks project. Creates the agent folder, app resource, experiment, and manifest entry. Triggers on "add agent", "new agent", "create another agent", "add-agent".

Add a custom LangGraph supervisor agent that routes across an AgentOps Stacks project's agents, scaffolded into databricks.yml and the manifest so it rides the same eval-gate + dev/staging/prod lifecycle as every other agent. First checks whether a supervisor…

Comprehensive AgentOps reference and advisory guide, distilled from Databricks' The Big Book of AgentOps (Rao, Choo, Wulffert, Baur; Jul 2026) and kept current with platform state. Covers deploying quality, reliable, safe, observable agentic systems on…

Scaffold a new AgentOps Stacks project — a multi-agent LangGraph bundle (DAB) with shared components, per-agent Databricks Apps, evaluation, and CI/CD. Use when the user wants to start a new AI agent project on Databricks. Triggers on "scaffold a new agentops…

Operate and troubleshoot the Lakebase (Autoscaling Postgres) memory component in an AgentOps Stacks project — check endpoint health, mint credentials, inspect or clear checkpointer (short-term) and store (long-term) tables, manage the Postgres schema. Use…

Analyze Enterprise Analytics & BI adoption from system.query.history — active users, query volumes, warehouse utilization, and slow-query hotspots. Ask me which dashboards/warehouses are busiest or where query performance is degrading.

Track Advanced Analytics model performance and inference volumes from ML monitoring tables — accuracy/drift metrics by model and version, plus serving request volumes. Ask me which models are degrading or which endpoints are busiest.

Report Unity Catalog governance coverage — table comments, ownership, tags, and column documentation — from system.information_schema. Ask me which tables lack owners, descriptions, or classification tags across a catalog.

Recommend Databricks cost-optimization actions for the FinOps/platform team — commitment/discount coverage, budget variance, and forecast-vs-actual DBU spend from system.billing.usage and system.billing.list_prices.

Discover data across Unity Catalog — find tables by name or topic, inspect columns and comments, and trace where data lives — from system.information_schema. An enterprise-wide skill for locating the right table before you query it, in any workspace.

A personal sandbox skill used as the annotated authoring template for this reference implementation. It shows the agentskills.io-recommended SKILL.md structure. Use it when learning how to author a new skill.

Analyze Financial Applications budget-vs-actual variance by cost center and account from aggregate financial ledger tables. Ask me where spend is over or under budget, variance trends, or the biggest variance drivers.

Analyze Databricks DBU cost trends, identify top-spending jobs, and recommend cost optimizations using system.billing.usage and system.lakeflow.jobs. Ask me about spend by SKU, job-level costs, or DBU consumption trends over time.

Translate a source's parsed inventory into Databricks IR (intermediate representation): run deterministic translators for known types, then agentic (LLM-assisted) translation for the gaps. Phase 2 of the flowx migration workflow; routes to a source-specific…

Help the user get a copy of their source orchestrator's pipeline definitions before discover: for Azure Data Factory, by using the factory's Git repository, downloading the ARM template from ADF Studio, or pulling it live through the Azure API (run by the…

Estimate what an Azure Data Factory estate costs today: a live scan of the factories' recent pipeline and activity runs, priced at Azure list rates and reconciled against Azure-billed actuals from Cost Management. Writes a cost report under metadata/tco/.…

Prepare flowx to run its phases (discover, convert, package, migrate). In Databricks Genie Code this deploys the phases as an MCP server (a Databricks App) and creates NO virtual environment — all code runs through MCP. Everywhere else it provisions a Python…

Parse a source orchestrator's pipeline definitions (Azure Data Factory, Apache Airflow) into a typed inventory that classifies every task as deterministic, agentic, or unsupported. Phase 1 of the flowx migration workflow; routes to a source-specific guide.

End-to-end migration of a source orchestrator's pipelines (Azure Data Factory, Apache Airflow) to Databricks Lakeflow Jobs. Orchestrates discover, convert, and package phases in sequence.

Resolve source-reconciled Airflow leaf gaps through the fingerprint-bound flowx contract. Use after Airflow conversion emits PlaceholderActivity tasks and before packaging the reviewed report.

Deploy the per-pipeline Declarative Automation Bundles from a multi-pipeline flowx migration in dependency order, resolving cross-bundle job ids automatically. Local CLI only.

Reference for the Databricks MLOps Quickstart repo structure, parameterization contract, and Challenger/Champion conventions. Use whenever the user asks about repo layout, naming patterns, three-level Unity Catalog references, notebook widgets, MLflow…

Adapt the MLOps Quickstart Databricks Asset Bundle (databricks.yml, resources/*.yml) and CI/CD pipelines (Azure DevOps, GitHub Actions). Use when the user wants to rename the bundle, change catalog/schema per environment, add a staging target, configure…

Adapt the MLOps Quickstart data ingestion notebook from the Iris placeholder to a custom data source (cloud storage, JDBC, API, Delta table, etc.). Use when the user wants to ingest their own dataset, swap the source data, rename the feature table, add…

Adapt the MLOps Quickstart inference notebooks (batch and realtime / serving endpoint) to a custom model and input schema. Use when the user wants to run batch scoring on new data, query the realtime serving endpoint, change the inference output table, or…

Adapt the MLOps Quickstart MLflow 3 deployment pipeline (evaluate → approve → deploy) to a custom model. Use when the user wants to change evaluation metrics or thresholds, customize the human-in-the-loop approval gate, adjust the serving endpoint…

Adapt the MLOps Quickstart model training notebook from the Iris classifier placeholder to a custom algorithm and problem type (regression, clustering, classification, forecasting). Use when the user wants to change the model, features, target, metrics, or…

Manage Python dependencies for the MLOps Quickstart via the root `requirements.txt` file. Use when the user wants to add, remove, or upgrade a Python package, pin or unpin a library version, fix a dependency conflict, or understand how notebooks install their…

Testing framework for evaluating Databricks skills. Use when building test cases for skills, running skill evaluations, comparing skill versions, or creating ground truth datasets with the Generate-Review-Promote (GRP) pipeline. Triggers include "test skill",…

Evaluates whether the agent selected appropriate MCP tools instead of shell workarounds. Load when the trace contains Bash tool calls that could have used MCP tools, or when evaluating tool call efficiency.

Databricks development guidance including Python SDK, Databricks Connect, CLI, and REST API. Use when working with databricks-sdk, databricks-connect, or Databricks APIs.

General response quality evaluation. Always applicable regardless of domain. Covers response structure, actionability, clarity, and hallucination detection.

SQL evaluation criteria for Databricks. Load when the trace contains execute_sql tool calls or SQL code in responses. Covers syntax validity, Unity Catalog patterns, and Databricks-specific SQL features.

Python development guidance with code quality standards, error handling, testing practices, and environment management. Use when writing, reviewing, or modifying Python code (.py files) or Jupyter notebooks (.ipynb files).

MANDATORY post-deployment verification. After any workspace + Unity Catalog deployment, you MUST run all THREE compute paths against a UC table — classic cluster, serverless SQL warehouse, serverless notebook job. Skipping any of the three is incomplete work.…

Manage Databricks identity and governance. Use when the user asks to create groups, users, service principals, set up RBAC, manage permissions, workspace assignments, or configure access control.

Provision and test Databricks workspaces. Use when the user asks to create a workspace, set up a new environment, provision infrastructure, bootstrap Databricks, test a workspace, verify a deployment, or run validation checks against a Databricks workspace.…

Set up private networking for Databricks. Use when the user asks about private link, hub-spoke architecture, NCC (Network Connectivity Configuration), serverless connectivity to private resources, VPC endpoints, or private endpoints.

Set up Unity Catalog on Databricks workspaces. Use when the user asks to configure Unity Catalog, create a metastore, set up catalogs, schemas, external locations, storage credentials, or configure data governance. Covers Azure, AWS, and GCP.

Configure Databricks workspace settings. Use when the user asks to create SQL warehouses, cluster policies, secret scopes, IP access lists, manage tokens, update workspace settings, or destroy infrastructure.

Use when the user wants their firsthand knowledge of building, deploying, or running this project with real customers, users, or developers turned into published documentation — e.g. "interview me for docs", "extract what I know", "help me document this so…

Audit and improve spec coverage for a given spec. Use when (1) a spec has low or 0% requirement coverage, (2) tests exist but lack @req tags, (3) code behaviors have drifted from the spec's success criteria, (4) you need to identify unspecified behaviors in…

You MUST use this before any creative work — creating features, building components, adding functionality, or modifying behavior. Starts from existing specs rather than scratch. Use when the user asks to build, add, change, or design anything, even if it…

Code verification and testing for the Human Evaluation Workshop. Use when (1) running tests after code changes, (2) writing new unit tests (pytest/vitest), (3) writing E2E tests with Playwright/TestScenario, (4) debugging test failures, (5) understanding what…

Use when you have a spec or requirements for a multi-step task, before touching code. Creates spec-linked implementation plans with TDD steps, exact file paths, and spec coverage tracking. Use this after brainstorming, when a user says 'plan this', 'how…

MLflow 3 GenAI evaluation for agent development. Use when (1) writing mlflow.genai.evaluate() code, (2) creating @scorer functions, (3) building evaluation datasets from traces, (4) using built-in scorers (Guidelines, Correctness, Safety,…

Add or modify a BFF (Backend-For-Frontend) aggregation route in hc-portal. Use when the user asks to "add a BFF endpoint", "join data across services", "compose a view that needs patient + appointment + lab", "fan out to multiple services", "build a portal…

Deploy services to dev/test/prod via Databricks Asset Bundles. Use when the user asks to "deploy", "promote", "deploy to dev/test/prod", "run bundle deploy", "validate the bundle", "add a resource to the bundle", "set a DAB variable", or anything about DAB…

Apply GitFlow rules to this repo and map them onto the CI/CD pipelines. Use when the user asks to "open a PR", "cut a release", "create a hotfix", "what branch should this go on", "promote to test/prod", "what does pr-validate do", or anything about the four…

Manage Lakebase database branches that mirror code branches. Use when the user asks to "create a Lakebase branch", "spin up a feature DB", "branch the database for this feature", "tear down feature branch", "list active branches", or anything about Lakebase…

Scaffold a new backend microservice in this healthcare reference architecture. Use when the user asks to "add a new service", "scaffold X service", "create a new microservice for Y", "bootstrap a new service", or "add a service called <name>". Wires APX…

Wire OBO (On-Behalf-Of) authentication into a service or BFF route. Use when the user asks to "add a route", "wire OBO", "set up authentication", "connect to Lakebase as the user", "extract user identity", or anything about user-passthrough auth. Enforces the…