databricks-solutions

Scaffold new Databricks AppKit applications using the Databricks CLI and Agent Skills. Creates blank or plugin-enabled AppKit projects (Lakebase, Analytics, Genie, Files). Use when asked to create a Databricks app, scaffold an AppKit project, bootstrap a new app, or set up a full-stack TypeScript Databricks application. Triggers on "create app", "new app", "scaffold", "AppKit", "databricks app", "blank app", "bootstrap app", "init app". To add a plugin to an existing app, use the 04-appkit-plugin-add skill instead.

Entry-point navigator for Databricks AppKit + Lakebase development. Routes tasks to the correct skill based on keyword detection: scaffolding, plugin integration, feature building, or deployment. This is a routing skill -- it does not generate code. It directs the agent to the correct specialized skill. Use this skill as the starting point for any AppKit-related task. Triggers on "AppKit", "Lakebase app", "Databricks app", "build app", "deploy app", "scaffold app", "add plugin", "AppKit project", "agent endpoint", "wire agent", "serving plugin", "agent app proxy", "chat history", "feedback".

Build full-stack UI and backend features on a Databricks AppKit project from a PRD or feature spec. Covers SQL query design, type generation, React frontend with AppKit UI components, backend plugin wiring, and distinctive visual design. Use when asked to implement a UI, build features from a PRD, create pages or dashboards, add components, develop the frontend/backend of an existing AppKit app, build with mock data, or use static data for visual prototyping. Triggers on "build UI", "implement PRD", "create dashboard", "add page", "build features", "implement design", "create components", "build app from PRD", "develop frontend", "mock data", "static data", "two-phase data".

Deploy a Databricks AppKit application to Databricks Apps. Covers config validation, build verification, deployment, UI verification, error diagnosis with fix loop, and workspace app limit handling. Use when asked to deploy an AppKit app, push to production, ship the app, or troubleshoot a failed deploy. Triggers on "deploy app", "push to databricks", "ship app", "deploy appkit", "databricks apps deploy", "fix deploy error", "app won't start".

Add plugins to an existing Databricks AppKit project. Covers Lakebase (PostgreSQL), Analytics (SQL queries + dashboards), Genie (natural language AI/BI), Files (UC Volumes), and Serving (Model Serving / Agent endpoints). Guides through plugin registration, environment variables, app.yaml configuration, and frontend integration. Use when asked to add a plugin to an existing app, integrate Lakebase, add analytics, connect a Genie space, enable file uploads, add a serving endpoint, or extend an AppKit project with new capabilities. Triggers on "add plugin", "add lakebase", "add analytics", "add genie", "add files plugin", "add serving", "add agent endpoint", "integrate postgres", "add database", "add dashboards", "add file browser", "extend app", "connect genie space", "model serving plugin". Not for creating a Lakebase project or managing Lakebase via CLI -- use the databricks-lakebase agent skill for that.

Wire a Lakebase PostgreSQL backend into an existing AppKit project. Covers database schema design from a PRD, idempotent DDL, Express API routes with mock fallback, React data hooks, and local testing. PRD-independent patterns that apply to any AppKit + Lakebase app. Use after registering the Lakebase plugin via 04-appkit-plugin-add. Triggers on "wire lakebase", "lakebase backend", "CRUD API", "lakebase tables", "DDL", "database schema design", "useLakebaseData", "mock fallback", "ConnectionStatus", "replace mock data", "connect frontend to backend", "API-backed data", "replace static data with database".

Wire a Databricks Model Serving or Agent endpoint into an existing AppKit project using the Serving plugin. Covers endpoint registration, app.yaml resource binding, streaming and invoke React hooks, conversation state management, agent response mapping, and server-side proxy patterns. PRD-independent patterns that apply to any AppKit + Serving app. Use after registering the Serving plugin via 04-appkit-plugin-add. Triggers on "wire agent", "agent endpoint", "serving plugin", "agent UI", "chat interface", "connect agent", "model serving", "useServingStream", "useServingInvoke", "agent chat", "wire serving", "serving wiring".

Wire an AppKit frontend (Databricks App) to a separate Agent App backend (Databricks App hosting mlflow.genai.agent_server or ResponsesAgent) with OBO token forwarding via x-forwarded-access-token, app-to-app service principal auth, dual-format SSE streaming, and the /api/chat contract. Use this skill when the agent is deployed as its own Databricks App — not as a Model Serving endpoint. Triggers on "agent app backend", "two apps", "appkit plus agent app", "proxy agent app", "agent-backend resource", "x-forwarded-access-token", "migrate agent to apps", "agent app proxy".

Create and configure Declarative Automation Bundles (formerly Asset Bundles) with best practices for multi-environment deployments (CICD). Use when working with: (1) Creating new DAB projects, (2) Adding resources (dashboards, pipelines, jobs, alerts), (3) Configuring multi-environment deployments, (4) Setting up permissions, (5) Deploying or running bundle resources

Patterns and best practices for Lakebase Provisioned (Databricks managed PostgreSQL) for OLTP workloads. Use when creating Lakebase instances, connecting applications or Databricks Apps to PostgreSQL, implementing reverse ETL via synced tables, storing agent or chat memory, or configuring OAuth authentication for Lakebase.

Testing framework for evaluating Databricks skills. Use when building test cases for skills, running skill evaluations, comparing skill versions, or creating ground truth datasets with the Generate-Review-Promote (GRP) pipeline. Triggers include "test skill", "evaluate skill", "skill regression", "ground truth", "GRP pipeline", "skill quality", and "skill metrics".

Builds Databricks applications. Prefers AppKit (TypeScript + React SDK) for new apps; falls back to Python frameworks (Dash, Streamlit, Gradio, Flask, FastAPI, Reflex) when Python is required. Handles OAuth authorization, app resources, SQL warehouse and Lakebase connectivity, model serving, foundation model APIs, and deployment. Use when building web apps, dashboards, ML demos, or REST APIs for Databricks, or when the user mentions AppKit, Streamlit, Dash, Gradio, Flask, FastAPI, Reflex, or Databricks app.

Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).

Databricks development guidance including Python SDK, Databricks Connect, CLI, and REST API. Use when working with databricks-sdk, databricks-connect, or Databricks APIs.

Patterns and best practices for Lakebase Autoscaling (next-gen managed PostgreSQL). Use when creating or managing Lakebase Autoscaling projects, configuring autoscaling compute or scale-to-zero, working with database branching for dev/test workflows, implementing reverse ETL via synced tables, or connecting applications to Lakebase with OAuth credentials.

Create Databricks AI/BI dashboards. Use when creating, updating, or deploying Lakeview dashboards. CRITICAL: You MUST test ALL SQL queries via execute_sql BEFORE deploying. Follow guidelines strictly.

Build PWAF-compliant ISV integrations with Databricks: OAuth, telemetry (User-Agent), Unity Catalog, JDBC, SDK, SQL drivers, REST API, Databricks Connect.

PWAF-compliant Python SQL Connector (databricks-sql-connector): PAT, OAuth M2M, OAuth U2M (custom OAuth app PKCE + token-env), credentials_provider patterns, error handling, retry logic. Use when building Python integrations that run SQL queries via a Databricks SQL warehouse.

Add a Databricks connector to an existing project that has no Databricks integration. Use when your product already exists and you want to add Databricks as a new data source or backend.

How to write a build_report.md for any PWAF connector. Captures skill traceability, sufficiency assessment, and test error/fix log. Use after implementing any connector.

How to structure a Databricks connector (REST or Python SDK): config, connect, operations, validation. Use when designing or building a new connector.

How to build and use a PWAF connector test runner (tests/run_all_tests.sh). Covers env isolation, auth types, single connector or single auth mode, parallel execution, browser tests, and report generation.

PWAF-compliant Databricks Connect (Python): PAT, OAuth M2M, OAuth U2M; serverless and classic compute. Use when building or testing Spark-over-Connect integrations.

PWAF-compliant Databricks SDK for Go (databricks-sdk-go): PAT, OAuth M2M, U2M token-env, U2M custom OAuth app (PKCE); useragent.WithProduct/WithPartner. Use when building or testing Go SDK workspace API integrations.

Run the full cold-start end-to-end deploy test in an isolated git worktree with always-safe teardown. Use when: (1) verifying the one-shot `make deploy` works on a FRESH workspace/catalog (cold-start), (2) the user says 'integration test', 'cold-start test', 'test the deploy end to end', or 'does deploy still work on a new workspace', (3) validating a change to deploy.sh / databricks.yml / the seed before opening a PR. PROJECT-AUTHORED (not a vendored skill).

Keep this repo's architecture diagrams and prose docs in sync with the code as it evolves. Use this whenever the user asks to update/refresh the architecture diagrams, sync docs with the code, audit docs for staleness, or check that docs/architecture.md still matches the implementation — and proactively after any significant change (a new graph node, tool, Lakebase schema, data-pipeline step, auth change, or frontend swap), since the diagrams and READMEs drift silently. Covers refreshing the four Mermaid diagrams in docs/architecture.md, reconciling the known stale-doc patterns, and verifying with greps + Mermaid validity.

Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).

Build apps on Databricks Apps platform. Use when asked to create dashboards, data apps, analytics tools, or visualizations. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.

Databricks CLI operations: auth, profiles, data exploration, and bundles. Contains up-to-date guidelines for Databricks-related CLI tasks.

Create, configure, validate, deploy, run, and manage Declarative Automation Bundles (DABs, formerly Databricks Asset Bundles). Use when working with Databricks resources via DABs including dashboards, jobs, pipelines, alerts, volumes, and apps.

Databricks Lakebase Postgres: projects, scaling, connectivity, Lakebase synced tables, and Data API. Use when asked about Lakebase databases, OLTP storage, or connecting apps to Postgres on Databricks.

Manage Databricks Model Serving endpoints via CLI. Use when asked to create, configure, query, or manage model serving endpoints for LLM inference, custom models, or external models.

Adapt the MLOps Quickstart Databricks Asset Bundle (databricks.yml, resources/*.yml) and CI/CD pipelines (Azure DevOps, GitHub Actions). Use when the user wants to rename the bundle, change catalog/schema per environment, add a staging target, configure permissions, switch from serverless to job clusters, or wire up secrets and branch triggers for deployment.

Adapt the MLOps Quickstart data ingestion notebook from the Iris placeholder to a custom data source (cloud storage, JDBC, API, Delta table, etc.). Use when the user wants to ingest their own dataset, swap the source data, rename the feature table, add bronze/silver/gold preprocessing stages, or extend `1_data_preprocessing/`.

Adapt the MLOps Quickstart inference notebooks (batch and realtime / serving endpoint) to a custom model and input schema. Use when the user wants to run batch scoring on new data, query the realtime serving endpoint, change the inference output table, or enable Lakehouse Monitoring on inference results.

Adapt the MLOps Quickstart MLflow 3 deployment pipeline (evaluate → approve → deploy) to a custom model. Use when the user wants to change evaluation metrics or thresholds, customize the human-in-the-loop approval gate, adjust the serving endpoint configuration, or wire a freshly registered model version to the deployment job.

Adapt the MLOps Quickstart model training notebook from the Iris classifier placeholder to a custom algorithm and problem type (regression, clustering, classification, forecasting). Use when the user wants to change the model, features, target, metrics, or registered model name, or when they ask how to fit a different ML problem into this template.

Manage Python dependencies for the MLOps Quickstart via the root `requirements.txt` file. Use when the user wants to add, remove, or upgrade a Python package, pin or unpin a library version, fix a dependency conflict, or understand how notebooks install their dependencies.

Reference for the Databricks MLOps Quickstart repo structure, parameterization contract, and Challenger/Champion conventions. Use whenever the user asks about repo layout, naming patterns, three-level Unity Catalog references, notebook widgets, MLflow experiment naming, or serving endpoint naming, or before adapting any notebook, job, or bundle in this repo.

Use when the user wants their firsthand knowledge of building, deploying, or running this project with real customers, users, or developers turned into published documentation — e.g. "interview me for docs", "extract what I know", "help me document this so others can run it without me".

Audit and improve spec coverage for a given spec. Use when (1) a spec has low or 0% requirement coverage, (2) tests exist but lack @req tags, (3) code behaviors have drifted from the spec's success criteria, (4) you need to identify unspecified behaviors in the codebase. Covers the full audit loop: analyze coverage -> tag existing tests -> identify spec gaps -> propose spec updates.

You MUST use this before any creative work — creating features, building components, adding functionality, or modifying behavior. Starts from existing specs rather than scratch. Use when the user asks to build, add, change, or design anything, even if it seems simple. Covers the full loop: find governing spec -> explore intent -> design within spec constraints -> transition to planning.

Code verification and testing for the Human Evaluation Workshop. Use when (1) running tests after code changes, (2) writing new unit tests (pytest/vitest), (3) writing E2E tests with Playwright/TestScenario, (4) debugging test failures, (5) understanding what to mock in E2E tests, (6) verifying a feature implementation. Covers the full test pyramid: unit tests -> integration tests -> E2E tests.

Use when you have a spec or requirements for a multi-step task, before touching code. Creates spec-linked implementation plans with TDD steps, exact file paths, and spec coverage tracking. Use this after brainstorming, when a user says 'plan this', 'how should we implement', or when you're about to start a multi-file feature. Covers the full loop: spec review -> file mapping -> task decomposition -> TDD steps -> coverage verification.

MLflow 3 GenAI evaluation for agent development. Use when (1) writing mlflow.genai.evaluate() code, (2) creating @scorer functions, (3) building evaluation datasets from traces, (4) using built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), (5) analyzing traces for latency/errors/architecture, (6) optimizing agent context/prompts/token usage, (7) debugging evaluation failures. Covers the full eval workflow: trace analysis -> dataset building -> scorer creation -> evaluation execution.

Scaffold a new AgentOps Stacks project — a multi-agent LangGraph bundle (DAB) with shared components, per-agent Databricks Apps, evaluation, and CI/CD. Use when the user wants to start a new AI agent project on Databricks. Triggers on "scaffold a new agentops project", "new DAB with CI/CD", "start a new Databricks AI project", "create agentops-stacks project".

Operate and troubleshoot the Lakebase memory component in an AgentOps Stacks project — check connection health, refresh credentials, inspect or clear checkpointer tables, manage the Postgres schema. Use when the user's agent is having memory/persistence issues.

Operate and troubleshoot the UC Functions tools component in an AgentOps Stacks project — register new functions, update EXECUTE grants, list available functions, re-run the registration job. Use when the user needs to add, update, or grant access to UC functions used by their agent.

Operate and troubleshoot the Vector Search component in an AgentOps Stacks project — check index status, trigger sync, test the retriever, update the DLT data pipeline. Use when the user needs to manage their VS index or debug retrieval quality.

Add a new agent to an existing AgentOps Stacks project. Creates the agent folder, app resource, experiment, and manifest entry. Triggers on "add agent", "new agent", "create another agent", "add-agent".

Guide an agentops-stacks project through its full production lifecycle — data preparation, agent development, evaluation gates, CI/CD promotion, and production monitoring — following the Single-Account Single-Agent pattern from the Big Book of AgentOps. Use after `databricks bundle init` has been run and `.agentops-stacks/manifest.yml` exists. Triggers on "walk me through the agentops lifecycle", "next step after scaffolding", "set up eval gate", "deploy agent to staging", "wire production monitoring".

Opinionated git-Lakebase branch-pairing workflows. Use when scaffolding a Lakebase-paired project, creating/deleting Lakebase branches in lockstep with git branches, diffing parent-aware schemas, opening or merging PRs that touch Lakebase, or running the same operations the lakebase-scm-extension exposes in VS Code.

Spec-First Test-Driven Development (SFTDD) with evolutionary design, against paired Lakebase branches: the design lane is Spec Driven Development (SDD) and the build lane is canonical Beck-style Test Driven Development (RED-GREEN-REFACTOR), composed with paired-branch primitives (cheap experiments, parent-aware schema diff, real per-branch databases). Spec-first means the spec is drafted, reviewed, and frozen at gates before any build cycle runs; evolutionary means the spec, the architecture, and the database all evolve increment over increment. Use when planning a new feature, running design-spec gates, running TDD cycles, comparing parallel experiments, or detecting workflow bad smells. Imports software-design-principles canon. Builds on lakebase-scm-workflows + lakebase-release-workflows.

Foundational engineering canon – SOLID, DRY, clean code, layered architecture, cross-cutting concerns, NFRs. Imported by workflow skills (lakebase-sftdd-workflows, lakebase-scm-workflows, lakebase-release-workflows). Use when designing a module, reviewing a PR, planning a refactor, mapping cross-cutting concerns to layers, or arguing about API shape.

System-level engineering canon, the architecture counterpart to software-design-principles. Layered architecture + dependency direction, ports and adapters (repository / ORM-as-adapter), the twelve-factor app for cloud-native development, evolutionary architecture with fitness functions, and evolutionary database design (schema that evolves by migration on paired branches). Imported by workflow skills (lakebase-sftdd-workflows, lakebase-scm-workflows, lakebase-release-workflows). Use when: shaping a system boundary, deciding what is a backing service, mapping config to the environment, or authoring the fitness functions that keep the architecture honest as it evolves.

Opinionated branching + release methodology for Lakebase-paired projects. Use when designing a project's branch layout, cutting a release candidate, promoting between long-running tiers, rolling back, or asking 'where should this work happen?' Encodes the prod / staging / {feature,test,uat,perf} default and the N-tier-capable cut-RC / regression-test / cut-backup / migrate release flow.

Experience-level engineering canon, the UI/UX counterpart to software-design-principles (code-level) and architectural-design-principles (system-level). Usability heuristics, visual hierarchy, accessibility, interaction and feedback, information architecture, design systems and tokens, and testable UI. Imported by the lakebase-sftdd-workflows UX Designer (authoring design-guide.{md,json} + ia.md and the adherence gate) and by the Driver building UI. Use when: shaping a design guide or information architecture, reviewing a user-facing surface, choosing a UI framework, or making the UI testable.

Add or modify a BFF (Backend-For-Frontend) aggregation route in hc-portal. Use when the user asks to "add a BFF endpoint", "join data across services", "compose a view that needs patient + appointment + lab", "fan out to multiple services", "build a portal route", or anything about cross-service aggregation. Enforces the "BFF is the only joiner" rule and the canonical concurrent-fetch / token-forwarding patterns.

Deploy services to dev/test/prod via Databricks Asset Bundles. Use when the user asks to "deploy", "promote", "deploy to dev/test/prod", "run bundle deploy", "validate the bundle", "add a resource to the bundle", "set a DAB variable", or anything about DAB targets, profiles, or app deployment. Codifies the per-service resource-include layout, the target → profile mapping, the app → Lakebase + app → app resource wiring, and the verbs to use at each step.

Apply GitFlow rules to this repo and map them onto the CI/CD pipelines. Use when the user asks to "open a PR", "cut a release", "create a hotfix", "what branch should this go on", "promote to test/prod", "what does pr-validate do", or anything about the four GitHub Actions workflows or the branch → environment mapping. Codifies the branch protection rules, naming conventions, and the deploy triggers.

Manage Lakebase database branches that mirror code branches. Use when the user asks to "create a Lakebase branch", "spin up a feature DB", "branch the database for this feature", "tear down feature branch", "list active branches", or anything about Lakebase branch lifecycle. Implements the per-feature-branch model where each code branch has a matching Lakebase branch used for both local dev and PR CI.

Scaffold a new backend microservice in this healthcare reference architecture. Use when the user asks to "add a new service", "scaffold X service", "create a new microservice for Y", "bootstrap a new service", or "add a service called <name>". Wires APX project, app.yaml, Lakebase project, DAB resource block, BFF client stub, tests, CODEOWNERS, and migrations directory consistent with the rest of the monorepo.

Wire OBO (On-Behalf-Of) authentication into a service or BFF route. Use when the user asks to "add a route", "wire OBO", "set up authentication", "connect to Lakebase as the user", "extract user identity", or anything about user-passthrough auth. Enforces the canonical pattern where every Postgres connection is opened with the calling user's OAuth credential and Unity Catalog enforces access at the data layer.

Package and deploy Databricks Asset Bundles with proper parameterization, multi-environment support, and serverless compute. Handles project structure, databricks.yml generation, validation, and deployment. Use when packaging tested code for production, deploying pipelines, or managing multi-environment deployments.

Production data engineering pipelines following medallion architecture (Bronze/Silver/Gold layers) with data ingestion, transformation, quality checks, Delta Lake optimization, and orchestration. Use when building ETL pipelines, medallion architecture, data lakes, or data transformation workflows.

End-to-end machine learning pipelines on Databricks including data exploration, feature engineering, model training with hyperparameter optimization, MLflow experiment tracking, model registration to Unity Catalog, and deployment as DABs. Use when building ML workflows, training models, or deploying ML pipelines.

Execute code on Databricks clusters using MCP Command Execution API. Supports stateless quick validation and stateful iterative development. Use when testing Python/SQL code on clusters, debugging pipelines, or validating transformations.

Manage Unity Catalog resources including catalogs, schemas, and tables. Handles discovery, creation, updates, and deletions with proper naming conventions and governance. Use when exploring catalogs, creating schemas, managing tables, or setting up data governance.

Translate parsed ADF pipeline AST into Databricks IR (intermediate representation). Runs deterministic translators for known activity types, then performs agentic (LLM-assisted) translation for the remaining gaps.

Load and parse Azure Data Factory pipeline definitions from Unity Catalog volumes or local directories. Produces a typed inventory that classifies every activity as deterministic, agentic, or unsupported.

End-to-end migration of Azure Data Factory pipelines to Databricks Lakeflow Jobs. Orchestrates discover, convert, and package phases in sequence.

Prepare flowx to run its phases (discover, convert, package, migrate). In Databricks Genie Code this deploys the phases as an MCP server (a Databricks App) and creates NO virtual environment — all code runs through MCP. Everywhere else it provisions a Python virtual environment for the CLI skills, and optionally a local (stdio) MCP server. Run this once before any other flowx skill, or whenever the environment is missing.

Generate Databricks Declarative Automation Bundles (DABs) from translated IR, including job definitions, notebooks, and setup scripts.

Use when debugging event loop hangs, task scheduling issues, call_soon vs call_soon_threadsafe confusion, Future callback timing, _enter_task/_leave_task conflicts, GIL contention patterns, per-step vs per-drive task context, uvloop compatibility problems, sniffio/anyio backend detection failures, streaming backpressure deadlocks, or native runtime context issues on the asyncio thread. Also use when verifying asyncio assumptions via quick Python one-liners.

Strict set of rules in terms of codebase development, design patterns, and best practices. Use when the user wants to develop a new feature or refactor existing code.

Quick reference for building full-stack Databricks Apps with apx (React + FastAPI). Use when working on apx projects, creating routes, adding components, or managing dev servers.

Create git commits with emoji prefixes and conventional commit messages. Use when the user wants to commit changes, push code, or asks to use "just pm" command.