project-planning

# Project Plan Methodology for Databricks Solutions ## Planning Mode **Default: Data Product Acceleration** — full breadth, all domains, all artifacts, **Gold layer required as planning basis**. This is the standard behavior described in this entire skill document below. **Workshop mode** is available for Learning & Enablement scenarios with hard artifact caps **and layer flexibility** — it can plan from the best available source layer (Gold, Gold design YAML, Silver, Bronze, or source CSV). Workshop mode is NEVER activated unless the user includes the **exact phrase** `planning_mode: workshop` in their prompt. > **Mode vs source layer:** `planning_mode` (acceleration | workshop) controls artifact caps and validation strictness. `planning_source.selected_layer` (gold | gold_design | silver | bronze | source_csv) records which input the plan was derived from and is set automatically by Phase 0 below. Acceleration mode FORCES `selected_layer = gold` (or `gold_design` only if explicitly allowed). Workshop mode picks the best available source via the Phase 0 priority order and stamps it onto every manifest. ### Mode Detection Rules 1. **Default is ALWAYS `acceleration`.** If the user does not explicitly declare workshop mode, use acceleration. 2. **Workshop mode requires EXPLICIT opt-in.** The user must include one of these EXACT phrases: - `planning_mode: workshop` - `"workshop mode"` - `"use workshop mode"` 3. **Do NOT infer workshop mode** from words like "small", "simple", "demo", "limited", "quick", "basic", "training", or "few". These are NOT triggers. A user may want a narrow-scope acceleration plan — that's still acceleration mode with fewer use cases. 4. **When in doubt, ask.** If the user's intent is ambiguous (e.g., "Create a plan for a workshop"), ask: *"Would you like full Data Product Acceleration mode (default) or Workshop mode with limited artifacts? To use workshop mode, include `planning_mode: workshop` in your request."* 5. **Confirm mode at the start.** The first line of any plan output should state the active mode: - `**Planning Mode:** Data Product Acceleration (default)` - `**Planning Mode:** Workshop (explicit opt-in — artifact caps active)` 6. **When workshop mode is activated,** read `references/workshop-mode-profile.md` for artifact caps, phase scope, and selection criteria. Do NOT read that reference otherwise. 7. **Propagate mode to manifests.** Add `planning_mode: workshop` or `planning_mode: acceleration` to all generated manifest YAML files. Downstream orchestrators seeing `workshop` MUST NOT expand beyond the listed artifacts via self-discovery. ## Overview Comprehensive methodology for creating multi-phase project plans for Databricks data platform solutions. This skill combines interactive project planning with architectural methodology, including templates, worked examples, and quality standards. **Key Assumption (mode-aware):** - **Acceleration mode (default):** Planning starts AFTER Bronze ingestion AND Gold layer design are complete. Gold is the required planning basis. These are prerequisites, not phases. Phase 0 will stop with a remediation message if Gold is missing. - **Workshop mode (`planning_mode: workshop`):** Planning AND deployment are layer-agnostic. Phase 0 selects the highest-fidelity input present from: deployed Gold, Gold design YAML, deployed Silver, deployed Bronze, or a source schema CSV. The selected layer is stamped onto every manifest as `planning_source.selected_layer`. Workshop manifests built from Silver or Bronze are marked `implementation_readiness: workshop_deployable` — downstream stages (semantic-layer, observability, ml, genai-agents) deploy directly against the selected layer. Workshop manifests built from a source CSV are marked `implementation_readiness: workshop_draft` (planning contract only — no live tables to deploy against). `requires_gold_promotion` is an **advisory** field; it is recommended for production but never blocks deployment. ## When to Use This Skill Use this skill when: - Creating architectural plans for Databricks data platform projects - Building observability, analytics, or monitoring solutions - Planning multi-artifact solutions (TVFs, Metric Views, Dashboards, Genie Spaces, Alerts, ML Models) - Developing agent-based frameworks for platform management - Creating frontend applications for data platform interaction - Starting a new project after Gold layer is complete ## Idempotency Guard (Run FIRST) **Before regenerating plans, detect existing artifacts to avoid clobbering work-in-progress.** A common failure mode is regenerating `plans/` wholesale on a re-run and overwriting user edits to manifests, addendums, or the Use Case Catalog. ```python from pathlib import Path PLANS_DIR = Path("plans") if PLANS_DIR.exists() and any(PLANS_DIR.iterdir()): existing = sorted(p.relative_to(".") for p in PLANS_DIR.rglob("*") if p.is_file()) print("Existing plan artifacts detected:") for p in existing: print(f" {p} (mtime={Path(p).stat().st_mtime})") print( "\nHow would you like to proceed?\n" " - regenerate (DELETE and rebuild all plan files — destructive)\n" " - incremental (keep existing files, only emit MISSING artifacts)\n" " - skip (exit this orchestrator — recommended default)\n" ) ``` **Rules:** - **Default is `skip`.** If the user is silent or ambiguous, assume `skip` and exit with a summary of existing files. - `regenerate` must be explicit. Confirm the action ("I will delete N files under `plans/` — proceed?") before doing anything destructive. - `incremental` is the right choice when downstream orchestrators (semantic-layer, observability, ml, genai-agents) reported a missing manifest — only emit the missing manifest, not the whole tree. **Escape flag:** Users can set `planning_allow_overwrite: true` in their prompt to skip the idempotency check (equivalent to choosing `regenerate` without interactive confirmation). --- ## Quick Start (5 Minutes) ### Fast Track: Create Your Project Plan ```bash # 1. Verify prerequisites for your mode: # Acceleration (default): # - Bronze ingestion ✅ # - Silver DLT streaming ✅ # - Gold dimensional model ✅ (REQUIRED) # Workshop (planning_mode: workshop): # - At least ONE of: deployed Gold, gold_layer_design/yaml/, deployed Silver, # deployed Bronze, or data_product_accelerator/context/*.csv # - Phase 0 picks the highest-fidelity input automatically. # 2. Run this prompt with your project info: "Create a phased project plan for {project_name} with: - Planning assets: {n} tables (Gold/Silver/Bronze depending on what is available) - Use cases: {use_case_1, use_case_2, use_case_3, etc.} - Target audience: {executives, analysts, data scientists} - Agent domains: {domain1, domain2, domain3, domain4, domain5}" # 3. Output: Complete plan structure in plans/ folder # - Acceleration emits gold-dependency-manifest.yaml. # - Workshop emits gold-dependency-manifest.yaml OR source-dependency-manifest.yaml # depending on the selected planning source layer. ``` ### Key Decisions (Answer These First) | Decision | Options | Your Choice | |----------|---------|-------------| | Agent Domains | Derive from business questions (typically 2-5) | __________ | | Phase 1 Addendums | TVFs, Metric Views, Dashboards, Monitoring, Genie, Alerts, ML | __________ | | Phase 2 Scope | AI Agents (optional) or skip | __________ | | Phase 3 Scope | Frontend App (optional) or skip | __________ | | Genie Space Count | Based on asset count vs 25-asset limit (see Rationalization) | __________ | | Agent Architecture | Agents use Genie Spaces (recommended) or Direct SQL | __________ | | Agent-Genie Mapping | 1:1, consolidated, or unified (based on asset volume) | __________ | ## Working Memory Management This orchestrator spans 3 phases. To maintain coherence without context pollution: **After each phase, persist a brief summary note** capturing: - **Phase 1:** Domain list with Gold table mappings, addendum selections, business questions per domain, artifact count estimates - **Phase 2:** Plan document file paths, cross-references verified, total artifact counts by type - **Phase 3:** Manifest file paths (semantic-layer, observability, ml, genai-agents), validation results, summary counts **What to keep in working memory:** Current phase's template, domain list + artifact inventory, and previous phase's summary. Discard intermediate outputs — they are on disk. Read templates from `assets/templates/` and references just-in-time, not upfront. --- ## Step-by-Step Workflow ### Phase 0: Planning Source Discovery (MANDATORY, runs before Phase 1) This phase decides WHICH layer the plan will be derived from and stamps the answer onto every emitted manifest as `planning_source`. It runs in **both** modes; the only difference is which selections are allowed. #### Step 0.1 — Inventory available planning inputs Detect each potential planning source. Record presence/absence in working memory. ```python from pathlib import Path from databricks.sdk import WorkspaceClient def detect_planning_sources(catalog: str, user_schema_prefix: str) -> dict: """Return a dict describing every potential planning source that exists. Priority order (highest fidelity first): 1. deployed_gold — live tables in <catalog>.<prefix>_gold 2. gold_design — gold_layer_design/yaml/*.yaml authored, deployment may or may not be done 3. deployed_silver — live tables in <catalog>.<prefix>_silver 4. deployed_bronze — live tables in <catalog>.<prefix>_bronze 5. source_csv — data_product_accelerator/context/*.csv (last resort) """ w = WorkspaceClient() sources = {} for layer, schema in ( ("deployed_gold", f"{user_schema_prefix}_gold"), ("deployed_silver", f"{user_schema_prefix}_silver"), ("deployed_bronze", f"{user_schema_prefix}_bronze"), ): try: tables = list(w.tables.list(catalog_name=catalog, schema_name=schema)) sources[layer] = {"schema": f"{catalog}.{schema}", "table_count": len(tables)} if tables else None except Exception: sources[layer] = None yaml_dir = Path("gold_layer_design/yaml") if yaml_dir.exists() and any(yaml_dir.glob("*.yaml")): sources["gold_design"] = {"path": str(yaml_dir), "yaml_count": len(list(yaml_dir.glob("*.yaml")))} else: sources["gold_design"] = None csvs = list(Path("data_product_accelerator/context").glob("*.csv")) sources["source_csv"] = {"paths": [str(c) for c in csvs]} if csvs else None return sources ``` #### Step 0.2 — Select the planning source by mode | Mode | Allowed `selected_layer` values | Selection rule | |------|---------------------------------|----------------| | `acceleration` (default) | `deployed_gold`, `gold_design` | Pick `deployed_gold` if present; else `gold_design` ONLY when explicitly accepted; else **STOP** with a Gold-required remediation message. | | `workshop` | `deployed_gold`, `gold_design`, `deployed_silver`, `deployed_bronze`, `source_csv` | Pick the highest-priority source present. Never silently fall through to a lower layer when a higher one exists. | **Acceleration STOP message:** > Planning in acceleration mode requires the Gold layer. Run the Gold Layer Design and Setup skills first, or re-run with `planning_mode: workshop` to plan from a lower layer. **Workshop selection log (must be printed):** ``` Phase 0 — Planning source selected: <selected_layer> Available: deployed_gold=<bool>, gold_design=<bool>, deployed_silver=<bool>, deployed_bronze=<bool>, source_csv=<bool> Reason: highest-fidelity available input under workshop mode ``` #### Step 0.3 — Derive readiness markers Compute the readiness fields that every emitted manifest must include. `requires_gold_promotion` is **advisory only** — it is a hint for production hardening, never a deployment gate. ```python def readiness_for(selected_layer: str, mode: str) -> dict: if selected_layer == "deployed_gold": # Production-deployable from Gold. return {"implementation_readiness": "gold_ready", "requires_gold_promotion": False} if selected_layer == "gold_design": # Deployable once the Gold layer is provisioned. return {"implementation_readiness": "gold_design_only", "requires_gold_promotion": False} # Silver, Bronze, source CSV — workshop only. if mode != "workshop": raise SystemExit("Non-Gold planning sources are only allowed in workshop mode.") if selected_layer in {"deployed_silver", "deployed_bronze"}: # Workshop builds the semantic layer directly on top of Silver/Bronze. # Gold promotion is recommended for production but not required to deploy. return {"implementation_readiness": "workshop_deployable", "requires_gold_promotion": False} if selected_layer == "source_csv": # No live tables — planning contract only; downstream stages will not # attempt to deploy until at least one live layer exists. return {"implementation_readiness": "workshop_draft", "requires_gold_promotion": False} raise SystemExit(f"Unknown selected_layer={selected_layer!r}") ``` **Readiness state semantics:** | `implementation_readiness` | When | Downstream behavior | |---|---|---| | `gold_ready` | Acceleration or workshop on `deployed_gold` | Full production deploy | | `gold_design_only` | Acceleration or workshop on `gold_design` (Gold YAML, no live tables yet) | Deploy after Gold provisioning; live-catalog checks advisory | | `workshop_deployable` | Workshop on `deployed_silver` or `deployed_bronze` | Deploy semantic layer / Genie Spaces directly against the Silver or Bronze schema; Gold promotion is an advisory next step | | `workshop_draft` | Workshop on `source_csv` only | Planning contract only; downstream stages stop and ask for at least one live layer | #### Step 0.4 — Stamp `planning_source` onto every manifest Every manifest emitted by Phases 1–3 (semantic-layer, observability, ml, genai-agents, gold-dependency, source-dependency) MUST carry a top-level block: ```yaml planning_source: selected_layer: deployed_gold | gold_design | deployed_silver | deployed_bronze | source_csv schema: "<catalog>.<schema>" # e.g. main.acme_gold (omit/null for source_csv) source_yaml_dir: "gold_layer_design/yaml" # only when selected_layer = gold_design source_csv_paths: ["data_product_accelerator/context/<file>.csv"] # only for source_csv selected_at: "<ISO-8601 UTC>" implementation_readiness: gold_ready | gold_design_only | workshop_deployable | workshop_draft requires_gold_promotion: true | false # advisory only; never a deployment gate ``` Downstream orchestrators (semantic-layer, observability, ml, genai-agents) read these fields: - `gold_ready` / `gold_design_only` / `workshop_deployable` — proceed with deployment against the layer the manifest declares (`gold_schema` for Gold sources; `silver_schema` / `bronze_schema` for workshop deployments on Silver/Bronze). - `workshop_draft` (only emitted when `selected_layer = source_csv`) — stop before deployment; the plan is a contract only.