00-uc-resources-foundation

# Foundation Step 0: UC Resources (Schemas + Volumes) > **Cross-cutting.** This skill is consumed by **every** Foundation step and > agent track. It exists so that no downstream skill (KA, agent memory, tool > exports, benchmark persistence, prompt registry) has to repeat schema or > volume provisioning. If you find yourself writing `CREATE SCHEMA` or > `CREATE VOLUME` inside another skill, delegate to F0 instead. ## What This Skill Owns | Resource | Purpose | Default name | |---|---|---| | **Agent UC schema** | Holds agent-owned assets — KA source volume, tool output volume, benchmark tables, prompt registry pointers, memory tables | `${user_schema_prefix}_agent` | | **Ops UC schema** | Holds operational assets — MLflow OTeL trace tables (F2), monitoring tables (SDLC 07), eval-run rollups (SDLC 04) | `${user_schema_prefix}_ops` | | **`knowledge_sources` MANAGED volume** (in agent schema) | Source markdown / PDFs for Knowledge Assistants (F5) and any retrieval-style tool | `/Volumes/${catalog}/${agent_schema}/knowledge_sources` | | **`agent_outputs` MANAGED volume** (in agent schema) | CSV exports, charts, generated artifacts produced by `@function_tool` calls | `/Volumes/${catalog}/${agent_schema}/agent_outputs` | | **`signoffs` MANAGED volume** (in ops schema) | Stakeholder + engineering sign-off `decision.md` artifacts (SDLC 04b) — the durable, UC-governed location read by SDLC 06 promotion gates | `/Volumes/${catalog}/${ops_schema}/signoffs` | | **Custom volumes** (in agent or ops schema) | Anything declared in `state://AgentSpec.required_volumes[]` | `/Volumes/${catalog}/${schema}/${name}` | > **`signoffs` is generic.** F0 pre-creates the volume under a workshop-neutral > name (`signoffs`) so SDLC 04b never has to provision its own UC location and > SDLC 06 can resolve `/Volumes/${catalog}/${ops_schema}/signoffs/v<N>/decision.md` > deterministically. The volume name does **not** carry any use-case-specific > prefix (no `skyloyalty_signoffs`, no `${use_case_slug}_signoffs`); the > per-version directory `v<N>/` carries the disambiguation. Every operation is **idempotent** (`CREATE SCHEMA IF NOT EXISTS`, `CREATE VOLUME IF NOT EXISTS`, both as DDL — an existing object is a no-op). It is safe to call this skill from any prompt that needs to be sure these resources exist; the cost on a warm workspace is one round-trip per resource. ## What This Skill Does NOT Own - **Catalog creation.** A UC catalog must already exist; F0 reads `state://Resources.uc.catalog` and validates the caller has `USE CATALOG` + `CREATE SCHEMA` on it. - **UC table DDL.** Tables for OTeL traces, benchmarks, memory, or monitoring are owned by their respective consuming skills (F2, SDLC 02/04/07). - **Source content.** F0 creates an empty `knowledge_sources` volume; F5 (or any other consumer) is responsible for populating it with files. - **Bronze data.** Bronze schema + Delta tables are owned by the bronze layer setup skill ([`data_product_accelerator/skills/bronze/00-bronze-layer-setup`](../../../data_product_accelerator/skills/bronze/00-bronze-layer-setup/SKILL.md)). ## When to Use - **Always**, near the top of the prompt sequence — before F1 (MLflow env) so that anything downstream can assume schemas + volumes exist. - Whenever you add a new consuming skill that needs a new volume — extend `state://AgentSpec.required_volumes[]` rather than putting `CREATE VOLUME` inside the consuming skill. - After `bootstrap` when the resolved spec is available, but before any other Foundation step. ## Prerequisites | Requirement | How to Check | |---|---| | `state://Resources.uc.catalog` is resolved | Set by `vibecoding-state` `bootstrap` op | | `state://Resources.uc.user_schema_prefix` is resolved | Derived in `bootstrap` from the user identity + `use_case_slug` | | Workshop SQL warehouse is RUNNING | `state://Resources.warehouse_id` (set by Prompt 1 preflight) — needed for `CREATE SCHEMA` execution via `WorkspaceClient.statement_execution` | | `databricks-sdk >= 0.30` installed | `python -c "from databricks.sdk import WorkspaceClient; from databricks.sdk.service.sql import StatementState"` | | Caller has `USE CATALOG` + `CREATE SCHEMA` on the catalog | `databricks unity-catalog catalogs get $CATALOG --output json \| jq '.privileges'` | --- ## Inputs ```yaml # Resolved by the caller from state — F0 does not read state directly. uc_catalog: "string (required)" # e.g. "main" user_schema_prefix: "string (required)" # e.g. "jane_d_booking_app" agent_schema: "string (required)" # default: "${user_schema_prefix}_agent" ops_schema: "string (required)" # default: "${user_schema_prefix}_ops" warehouse_id: "string (required)" # SQL warehouse for DDL execution required_volumes: # optional — defaults below if not provided - { name: "knowledge_sources", schema: "agent", comment: "KA + retrieval source files" } - { name: "agent_outputs", schema: "agent", comment: "Tool-generated artifacts (CSV, charts)" } - { name: "signoffs", schema: "ops", comment: "SDLC 04b stakeholder + engineering signoff decision.md artifacts" } # AgentSpec.required_volumes[] is appended verbatim ``` `agent_schema` / `ops_schema` may be overridden to any name that starts with `${user_schema_prefix}_`; a non-prefixed override raises `ValueError`, because RULE_10 sanctions in-session provisioning of the participant's own prefixed schemas only. ## Operations ```mermaid flowchart TD start["F0 invoked"] --> grant["Verify USE CATALOG + CREATE SCHEMA"] grant --> agentSch["CREATE SCHEMA IF NOT EXISTS<br/>${catalog}.${agent_schema}"] grant --> opsSch["CREATE SCHEMA IF NOT EXISTS<br/>${catalog}.${ops_schema}"] agentSch --> volLoop["For each required_volume:<br/>create MANAGED volume idempotently"] opsSch --> volLoop volLoop --> capture["Capture { agent_schema, ops_schema, uc_volumes{name → path} }"] ``` ### Reference implementation ```python import time from databricks.sdk import WorkspaceClient from databricks.sdk.service.sql import StatementState def provision_uc_resources( *, uc_catalog: str, user_schema_prefix: str, agent_schema: str, ops_schema: str, warehouse_id: str, required_volumes: list[dict] | None = None, ddl_timeout_s: int = 300, ) -> dict: w = WorkspaceClient() # F0 provisions only the participant's own prefixed schemas (RULE_10 foundation carve-out). suffixes = {} for role, schema in (("agent", agent_schema), ("ops", ops_schema)): if not schema.startswith(f"{user_schema_prefix}_") or schema == f"{user_schema_prefix}_": raise ValueError( f"{role}_schema {schema!r} must start with '{user_schema_prefix}_': RULE_10 " "sanctions in-session provisioning of the participant's own prefixed schemas only" ) suffixes[role] = schema.removeprefix(f"{user_schema_prefix}_") def run_ddl(statement: str) -> None: """Run one DDL statement to a terminal state; raise unless it SUCCEEDED (fail closed).""" resp = w.statement_execution.execute_statement( warehouse_id=warehouse_id, statement=statement, wait_timeout="30s", ) deadline = time.monotonic() + ddl_timeout_s while resp.status.state in (StatementState.PENDING, StatementState.RUNNING): if time.monotonic() > deadline: w.statement_execution.cancel_execution(resp.statement_id) raise TimeoutError(f"DDL still {resp.status.state.value} after {ddl_timeout_s}s: {statement}") time.sleep(2) resp = w.statement_execution.get_statement(resp.statement_id) if resp.status.state != StatementState.SUCCEEDED: # FAILED / CANCELED / CLOSED error = resp.status.error.message if resp.status.error else "no error message" raise RuntimeError(f"DDL {resp.status.state.value}: {error} ({statement})") # 1. Schemas (DDL via statement execution against the workshop warehouse) for suffix in suffixes.values(): run_ddl( f"CREATE SCHEMA IF NOT EXISTS {uc_catalog}.{user_schema_prefix}_{suffix} " f"COMMENT 'F0-managed: agent or ops assets'" ) # 2. MANAGED volumes (DDL IF NOT EXISTS through the same warehouse) defaults = [ {"name": "knowledge_sources", "schema": "agent", "comment": "KA + retrieval source files"}, {"name": "agent_outputs", "schema": "agent", "comment": "Tool-generated artifacts (CSV, charts)"}, {"name": "signoffs", "schema": "ops", "comment": "SDLC 04b stakeholder + engineering signoff decision.md artifacts"}, ] volumes_spec = (required_volumes or []) + defaults seen, paths = set(), {} for v in volumes_spec: suffix = suffixes["agent" if v["schema"] == "agent" else "ops"] target_schema, name = f"{user_schema_prefix}_{suffix}", v["name"] key = (target_schema, name) if key in seen: continue seen.add(key) comment = v.get("comment", "F0-managed volume").replace("'", "\\'") run_ddl( f"CREATE VOLUME IF NOT EXISTS {uc_catalog}.{user_schema_prefix}_{suffix}.{name} " f"COMMENT '{comment}'" ) # raises unless SUCCEEDED, so only confirmed volumes are captured paths[name] = f"/Volumes/{uc_catalog}/{target_schema}/{name}" return { "agent_schema": agent_schema, "ops_schema": ops_schema, "uc_volumes": paths, } ``` The skill caller passes the result back to `vibecoding-state` `exit` so downstream skills can read `state://Resources.uc_volumes.knowledge_sources` etc. --- ## Outputs (handoff) | Key | Consumer | Value | |---|---|---| | `agent_schema` | F2, F5, SDLC 02/04, Track A 05 (memory) | `${user_schema_prefix}_agent` | | `ops_schema` | F2 (OTeL tables), SDLC 07 (monitoring), SDLC 04b (signoffs) | `${user_schema_prefix}_ops` | | `uc_volumes.knowledge_sources` | F5 (Step 5_0 — branch B/C upload), F3 (Vector Search MCP source dir) | `/Volumes/${catalog}/${agent_schema}/knowledge_sources` | | `uc_volumes.agent_outputs` | Track A 03 `csv_export` `@function_tool`, any code-interpreter output | `/Volumes/${catalog}/${agent_schema}/agent_outputs` | | `uc_volumes.signoffs` (alias `signoff_volume_path`) | SDLC 04b (writes `decision.md`), SDLC 06 (reads it as the promotion gate) | `/Volumes/${catalog}/${ops_schema}/signoffs` | | `uc_volumes.<custom>` | Whichever consuming skill declared the volume | `/Volumes/${catalog}/${schema}/${name}` | --- ## Validation Gate - [ ] `${uc_catalog}.${agent_schema}` exists (`SHOW SCHEMAS IN ${uc_catalog} LIKE '${agent_schema}'`) - [ ] `${uc_catalog}.${ops_schema}` exists - [ ] Every volume in `required_volumes` is listable via `databricks volumes list ${uc_catalog} ${schema} --output json | jq '.volumes[] | select(.name=="<name>")'` - [ ] The generic `signoffs` volume in `${ops_schema}` is present (consumed by SDLC 04b / SDLC 06 — name is workshop-neutral) - [ ] `agent_schema`, `ops_schema`, the full `uc_volumes` map, and `signoff_volume_path` are captured in state --- ## Delegation Map | Need | Delegate To | |---|---| | Bronze schema + Delta tables | [`bronze/00-bronze-layer-setup`](../../../data_product_accelerator/skills/bronze/00-bronze-layer-setup/SKILL.md) | | OTeL trace tables in `${ops_schema}` | [F2](../02-experiment-tracing-and-uc-storage/SKILL.md) | | KA source markdown into `knowledge_sources` | [F5 Step 5_0](../05-knowledge-assistant/SKILL.md) — branch (A) skip if non-empty / (B) upload `ka_source` / (C) auto-generate from PRD | | Lakebase memory tables | [Track A 05](../../tracks/A-custom-agent-apps/05-memory-and-state/SKILL.md) | | Benchmark tables in `${agent_schema}` | [SDLC 02](../../sdlc/02-evaluation-datasets/SKILL.md) |