# Genie Space Validator
```
Evaluates a Databricks Genie Space configuration and its underlying Unity
Catalog data against best practices. Produces a prioritized Markdown report
with actionable findings using a LangGraph pipeline backed by a Databricks
Model Serving LLM.
```
## Video Overview
https://github.com/user-attachments/assets/b047ee34-1a45-471d-8074-79e7ad251af4
## How It Works
The validator runs a [LangGraph](https://github.com/langchain-ai/langgraph) pipeline:
```
START ──┬── analyze data & config ─┐
├── fetch best practices ──┼── evaluate ── generate report ── END
└── run benchmarks ────────┘
```
1. **Analyze Genie data & config** (`load_state`): loads the space config and Unity Catalog metadata (live extraction or cached JSON files). Live extraction can take several minutes on large datasets.
2. **Fetch best practices** (`fetch_best_practices`, optional): pulls current Databricks docs via Jina Reader. Skippable; the built-in reference (`genie_validator/prompts/best_practices.md`) is always the baseline.
3. **Run benchmarks** (`run_benchmarks`, optional, live only): runs the space's benchmark questions through the Genie Eval API, in parallel with the analysis, and feeds the GOOD/BAD/NEEDS_REVIEW results into the evaluation. Requires `databricks-sdk >= 0.102.0`.
4. **Evaluate** (`evaluator`): an LLM scores the space across the tracks below and returns structured findings (Pydantic).
5. **Generate report** (`report_generator`): formats the findings into a Markdown report grouped by severity.
## Project Structure
```
genie_validator/
prompts/
best_practices.md # Built-in evaluation reference (editable)
org_conventions.md # Org-specific conventions (editable)
evaluator_system.md # System prompt for the evaluator LLM
report_system.md # System prompt for the report generator
extractor.py # Live extraction from Genie API + Unity Catalog
benchmarks.py # Genie Eval API benchmark runner
graph.py # LangGraph StateGraph and run_validator() entry point
models.py # ValidatorState, Finding, EvaluationReport (Pydantic)
nodes.py # Node implementations
.claude/skills/
refresh-genie-best-practices/ # Claude Code skill to sync best_practices.md with Databricks docs
run_validator.ipynb # Databricks notebook entry point
space_configs/ # Cached JSON files (per space)
reports/ # Generated evaluation reports
```
## Installation
### Prerequisites
- A Databricks workspace with a **pro or serverless SQL warehouse**.
- An LLM endpoint accessible via Databricks Model Serving (default: `databricks-claude-opus-4-8`). The Claude Opus endpoints (`databricks-claude-opus-4-8` and `databricks-claude-opus-4-6`) are the validated ones.
- A Git folder in Databricks linked to this repository.
### Setup
1. Create a Git folder in Databricks pointing to this repo.
2. Open `run_validator.ipynb` in the Databricks notebook UI.
3. Attach a cluster (Python 3.12+).
### Configuration
Edit the configuration cell to match your space:
```python
SPACE_NAME = "my_space" # Short label, used for file names and report output
# --- Live mode (default): extract directly from Genie + Unity Catalog ---
SPACE_CONFIG_PATH = None
DATA_CONTEXT_PATH = None
SPACE_ID = "<your-space-id>" # hex ID from the Genie Space URL
# --- Offline mode: comment out the three lines above and uncomment below ---
# SPACE_CONFIG_PATH = f"space_configs/{SPACE_NAME}_space_config.json"
# DATA_CONTEXT_PATH = f"space_configs/{SPACE_NAME}_data_context.json"
# SPACE_ID = None
LLM_ENDPOINT = "databricks-claude-opus-4-8"
SKIP_FETCH_BEST_PRACTICES = False # Set True on restricted-egress clusters (Jina Reader fetch is skipped, built-in reference is always used)
MAX_VIEW_DEF_TOKENS = 3000 # 0 = drop view defs, >0 = per-view token cap
# Optional benchmarks (live mode only; needs benchmark questions in the space)
RUN_BENCHMARKS = False
MAX_BENCHMARK_QUESTIONS = 5 # None runs all
```
**Live mode** (default): provide the `SPACE_ID` (hex ID from the Genie Space URL). The extractor queries the Genie API and Unity Catalog, then caches the results as JSON files for future offline runs.
**Offline mode**: switch the comment blocks above to point at cached JSON files under `space_configs/`. Use live mode at least once to generate them.
### Running
Execute the notebook cells in order:
| Step | Purpose |
| ---- | ------------------------------------------------------- |
| 1 | Install dependencies and restart Python |
| 2 | Set configuration variables |
| 3 | Verify the LLM endpoint is reachable |
| 4 | Run the evaluation pipeline |
| 5 | Save the report to `reports/{SPACE_NAME}_evaluation.md` |
### Local Development
```bash
# Requires Python 3.12+ and uv
uv sync
uv run python -c "from genie_validator.graph import run_validator; print('OK')"
# Optional: install the pre-commit hook (blocks accidental internal-mirror leaks in uv.lock)
uvx pre-commit install
```
The evaluation pipeline requires a Databricks LLM endpoint. Live extraction requires a Databricks cluster. Offline mode with cached JSONs works locally if the endpoint is reachable.
## Evaluation Tracks
The evaluator assesses each space across these tracks:
| Track | What it checks |
| ------------------------------------ | -------------------------------------------------------------------- |
| **A. Compliance** | Hard limits, descriptions, joins, instruction priorities, benchmarks |
| **B. Data Quality** | UC metadata: types, naming, PK/FK, sample value signals |
| **C. UC <-> Genie Cross-Validation** | Config references vs. actual schema; compensating controls |
| **D. Architecture** | Space focus, table consolidation, instruction mix |
| **E. Org Conventions** | Org-specific rules from `org_conventions.md` (naming, fiscal calendar, etc.) |
| **F. Benchmark Results** | When benchmarks ran: accuracy and failure patterns tied to config gaps |
| **G. Open-Ended Review** | Inconsistencies, orphaned references, security concerns |
Findings are classified by severity: **Critical** (must fix), **Recommended** (should fix), **Optional** (nice to have).
## Updating Best Practices
The built-in reference at `genie_validator/prompts/best_practices.md` is the primary source of evaluation rules. It is derived from:
- [Curate an effective Genie space](https://docs.databricks.com/aws/en/genie/best-practices)
- [Set up and manage a Genie space](https://docs.databricks.com/aws/en/genie/set-up)
- [Build a knowledge store](https://docs.databricks.com/aws/en/genie/knowledge-store)
- [Use parameters in SQL queries](https://docs.databricks.com/aws/en/genie/query-params)
- [Use trusted assets](https://docs.databricks.com/aws/en/genie/trusted-assets)
- [Use benchmarks](https://docs.databricks.com/aws/en/genie/benchmarks)
When Databricks updates these docs, review the changes and update `best_practices.md` manually. The evaluator system prompt (`evaluator_system.md`) references sections by number, so preserve the section structure when editing. The `Last updated:` date at the top of `best_practices.md` is parsed and shown in the report.
To make this easier, the repo ships a Claude Code skill: run `/refresh-genie-best-practices` (in Claude Code, from the repo root). It fetches the current Genie docs, compares them against the local reference, proposes a prioritized list of in-scope updates for your approval, applies the ones you accept while keeping the section structure and date line intact, and flags any conflicts with `org_conventions.md`. The skill lives at `.claude/skills/refresh-genie-best-practices/` and is auto-discovered on clone.
To add organization-specific rules (naming conventions, fiscal calendar, currency or timezone defaults), edit `genie_validator/prompts/org_conventions.md`. It is injected into the evaluator as a separate section, so the same agent can be tailored per organization.
## How to get help
Databricks support doesn't cover this content. For questions or bugs, please open a GitHub issue and the team will help on a best effort basis.
## License
© 2026 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License [[https://databricks.com/db-license-source]](https://databricks.com/db-license-source]). All included or referenced third party libraries are subject to the licenses set forth below.
| library | description | license | source |
| -------------------- | ------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------- |
| langgraph | Graph orchestration for LLM workflows | MIT | [https://github.com/langchain-ai/langgraph](https://github.com/langchain-ai/langgraph) |
| databricks-langchain | Databricks integrations for LangChain | Apache-2.0 | [https://github.com/databricks/databricks-langchain](https://github.com/databricks/databricks-langchain) |
| databricks-sdk | Databricks SDK for Python | Apache-2.0 | [https://github.com/databricks/databricks-sdk-py](https://github.com/databricks/databricks-sdk-py) |
| pydantic | Data validation via Python type hints | MIT | [https://github.com/pydantic/pydantic](https://github.com/pydantic/pydantic) |
| requests | HTTP library | Apache-2.0 | [https://github.com/psf/requests](https://github.com/psf/requests) |
| tiktoken | BPE tokenizer | MIT | [https://github.com/openai/tiktoken](https://github.com/openai/tiktoken) |