project-planning

name	project-planning
description	Create multi-phase project plans for Databricks data platform solutions with Agent Domain Framework and Agent Layer Architecture. Includes interactive Quick Start with key decisions, industry-specific domain patterns, complete phase document templates (Use Cases, Agents, Frontend), Genie Space integration patterns, deployment order requirements, and worked examples. Supports both acceleration mode (plan on a completed Gold layer) and workshop mode (`planning_mode: workshop`) that plans from the best available layer with hard artifact caps. Use when planning any Databricks solution post-Gold layer — observability, analytics, agent-based frameworks, or multi-artifact projects.
clients	["ide_cli","genie_code"]
bundle_resource	none
deploy_verb	bundle_deploy
deploy_note	Design-phase skill: produces multi-phase plans, manifests, and addendums; it has no deployed resource of its own — downstream artifacts deploy later via `bundle deploy --target dev` (runDatabricksCli on Genie Code). B12: the Gold-gap STOP gate writes `plans/gold-gap-remediation.md` and halts before generating plans; workshop mode (`planning_mode: workshop`) applies hard artifact caps. On Genie Code, write all generated plans/manifests under the cloned repo root (`{REPO_ROOT}` = `state_file_root` from `skills/vibecoding-state`, e.g. `plans/`), not a bare relative path — relative paths resolve against the page CWD (see `skills/genie-code-environment` §8).
coverage	full
metadata	{"author":"prashanth subrahmanyam","version":"2.0","domain":"planning","role":"orchestrator","pipeline_stage":5,"pipeline_stage_name":"planning","next_stages":["semantic-layer-setup"],"workers":[],"common_dependencies":["databricks-expert-agent","naming-tagging-standards"],"emits":["plans/use-case-catalog.md","plans/manifests/semantic-layer-manifest.yaml","plans/manifests/observability-manifest.yaml","plans/manifests/ml-manifest.yaml","plans/manifests/genai-agents-manifest.yaml","plans/manifests/gold-dependency-manifest.yaml","plans/manifests/source-dependency-manifest.yaml","plans/gold-gap-remediation.md","plans/source-gap-remediation.md"],"reads":["gold_layer_design/yaml/","gold_layer_design/erd_master.md","gold_layer_design/docs/BUSINESS_ONBOARDING_GUIDE.md","data_product_accelerator/context/*.csv"],"supported_modes":["acceleration","workshop"],"default_mode":"acceleration","last_verified":"2026-02-07","volatility":"low","upstream_sources":[]}

Project Plan Methodology for Databricks Solutions

Planning Mode

Default: Data Product Acceleration — full breadth, all domains, all artifacts, Gold layer required as planning basis. This is the standard behavior described in this entire skill document below.

Workshop mode is available for Learning & Enablement scenarios with hard artifact caps and layer flexibility — it can plan from the best available source layer (Gold, Gold design YAML, Silver, Bronze, or source CSV). Workshop mode is NEVER activated unless the user includes the exact phrase planning_mode: workshop in their prompt.

Mode vs source layer: planning_mode (acceleration | workshop) controls artifact caps and validation strictness. planning_source.selected_layer (gold | gold_design | silver | bronze | source_csv) records which input the plan was derived from and is set automatically by Phase 0 below. Acceleration mode FORCES selected_layer = gold (or gold_design only if explicitly allowed). Workshop mode picks the best available source via the Phase 0 priority order and stamps it onto every manifest.

Mode Detection Rules

1. Default is ALWAYS acceleration. If the user does not explicitly declare workshop mode, use acceleration.
2. Workshop mode requires EXPLICIT opt-in. The user must include one of these EXACT phrases:
   • planning_mode: workshop
   • "workshop mode"
   • "use workshop mode"
3. Do NOT infer workshop mode from words like "small", "simple", "demo", "limited", "quick", "basic", "training", or "few". These are NOT triggers. A user may want a narrow-scope acceleration plan — that's still acceleration mode with fewer use cases.
4. When in doubt, ask. If the user's intent is ambiguous (e.g., "Create a plan for a workshop"), ask: "Would you like full Data Product Acceleration mode (default) or Workshop mode with limited artifacts? To use workshop mode, include planning_mode: workshop in your request."
5. Confirm mode at the start. The first line of any plan output should state the active mode:
   • **Planning Mode:** Data Product Acceleration (default)
   • **Planning Mode:** Workshop (explicit opt-in — artifact caps active)
6. When workshop mode is activated, read references/workshop-mode-profile.md for artifact caps, phase scope, and selection criteria. Do NOT read that reference otherwise.
7. Propagate mode to manifests. Add planning_mode: workshop or planning_mode: acceleration to all generated manifest YAML files. Downstream orchestrators seeing workshop MUST NOT expand beyond the listed artifacts via self-discovery.

Overview

Comprehensive methodology for creating multi-phase project plans for Databricks data platform solutions. This skill combines interactive project planning with architectural methodology, including templates, worked examples, and quality standards.

Key Assumption (mode-aware):

• Acceleration mode (default): Planning starts AFTER Bronze ingestion AND Gold layer design are complete. Gold is the required planning basis. These are prerequisites, not phases. Phase 0 will stop with a remediation message if Gold is missing.
• Workshop mode (planning_mode: workshop): Planning AND deployment are layer-agnostic. Phase 0 selects the highest-fidelity input present from: deployed Gold, Gold design YAML, deployed Silver, deployed Bronze, or a source schema CSV. The selected layer is stamped onto every manifest as planning_source.selected_layer. Workshop manifests built from Silver or Bronze are marked implementation_readiness: workshop_deployable — downstream stages (semantic-layer, observability, ml, genai-agents) deploy directly against the selected layer. Workshop manifests built from a source CSV are marked implementation_readiness: workshop_draft (planning contract only — no live tables to deploy against). requires_gold_promotion is an advisory field; it is recommended for production but never blocks deployment.

When to Use This Skill

Use this skill when:

• Creating architectural plans for Databricks data platform projects
• Building observability, analytics, or monitoring solutions
• Planning multi-artifact solutions (TVFs, Metric Views, Dashboards, Genie Spaces, Alerts, ML Models)
• Developing agent-based frameworks for platform management
• Creating frontend applications for data platform interaction
• Starting a new project after Gold layer is complete

Idempotency Guard (Run FIRST)

Before regenerating plans, detect existing artifacts to avoid clobbering work-in-progress. A common failure mode is regenerating plans/ wholesale on a re-run and overwriting user edits to manifests, addendums, or the Use Case Catalog.

from pathlib import Path

PLANS_DIR = Path("plans")
if PLANS_DIR.exists() and any(PLANS_DIR.iterdir()):
    existing = sorted(p.relative_to(".") for p in PLANS_DIR.rglob("*") if p.is_file())
    print("Existing plan artifacts detected:")
    for p in existing:
        print(f"  {p}  (mtime={Path(p).stat().st_mtime})")
    print(
        "\nHow would you like to proceed?\n"
        "  - regenerate  (DELETE and rebuild all plan files — destructive)\n"
        "  - incremental (keep existing files, only emit MISSING artifacts)\n"
        "  - skip        (exit this orchestrator — recommended default)\n"
    )

Rules:

• Default is skip. If the user is silent or ambiguous, assume skip and exit with a summary of existing files.
• regenerate must be explicit. Confirm the action ("I will delete N files under plans/ — proceed?") before doing anything destructive.
• incremental is the right choice when downstream orchestrators (semantic-layer, observability, ml, genai-agents) reported a missing manifest — only emit the missing manifest, not the whole tree.

Escape flag: Users can set planning_allow_overwrite: true in their prompt to skip the idempotency check (equivalent to choosing regenerate without interactive confirmation).

---

Quick Start (5 Minutes)

Fast Track: Create Your Project Plan

# 1. Verify prerequisites for your mode:
#    Acceleration (default):
#      - Bronze ingestion ✅
#      - Silver DLT streaming ✅
#      - Gold dimensional model ✅ (REQUIRED)
#    Workshop (planning_mode: workshop):
#      - At least ONE of: deployed Gold, gold_layer_design/yaml/, deployed Silver,
#        deployed Bronze, or data_product_accelerator/context/*.csv
#      - Phase 0 picks the highest-fidelity input automatically.

# 2. Run this prompt with your project info:
"Create a phased project plan for {project_name} with:
- Planning assets: {n} tables (Gold/Silver/Bronze depending on what is available)
- Use cases: {use_case_1, use_case_2, use_case_3, etc.}
- Target audience: {executives, analysts, data scientists}
- Agent domains: {domain1, domain2, domain3, domain4, domain5}"

# 3. Output: Complete plan structure in plans/ folder
#    - Acceleration emits gold-dependency-manifest.yaml.
#    - Workshop emits gold-dependency-manifest.yaml OR source-dependency-manifest.yaml
#      depending on the selected planning source layer.

Key Decisions (Answer These First)

Decision	Options	Your Choice
Agent Domains	Derive from business questions (typically 2-5)	__________
Phase 1 Addendums	TVFs, Metric Views, Dashboards, Monitoring, Genie, Alerts, ML	__________
Phase 2 Scope	AI Agents (optional) or skip	__________
Phase 3 Scope	Frontend App (optional) or skip	__________
Genie Space Count	Based on asset count vs 25-asset limit (see Rationalization)	__________
Agent Architecture	Agents use Genie Spaces (recommended) or Direct SQL	__________
Agent-Genie Mapping	1:1, consolidated, or unified (based on asset volume)	__________

Working Memory Management

This orchestrator spans 3 phases. To maintain coherence without context pollution:

After each phase, persist a brief summary note capturing:

• Phase 1: Domain list with Gold table mappings, addendum selections, business questions per domain, artifact count estimates
• Phase 2: Plan document file paths, cross-references verified, total artifact counts by type
• Phase 3: Manifest file paths (semantic-layer, observability, ml, genai-agents), validation results, summary counts

What to keep in working memory: Current phase's template, domain list + artifact inventory, and previous phase's summary. Discard intermediate outputs — they are on disk. Read templates from assets/templates/ and references just-in-time, not upfront.

---

Step-by-Step Workflow

Phase 0: Planning Source Discovery (MANDATORY, runs before Phase 1)

This phase decides WHICH layer the plan will be derived from and stamps the answer onto every emitted manifest as planning_source. It runs in both modes; the only difference is which selections are allowed.

Step 0.1 — Inventory available planning inputs

Detect each potential planning source. Record presence/absence in working memory.

from pathlib import Path
from databricks.sdk import WorkspaceClient

def detect_planning_sources(catalog: str, user_schema_prefix: str) -> dict:
    """Return a dict describing every potential planning source that exists.

    Priority order (highest fidelity first):
      1. deployed_gold     — live tables in <catalog>.<prefix>_gold
      2. gold_design       — gold_layer_design/yaml/*.yaml authored, deployment may or may not be done
      3. deployed_silver   — live tables in <catalog>.<prefix>_silver
      4. deployed_bronze   — live tables in <catalog>.<prefix>_bronze
      5. source_csv        — data_product_accelerator/context/*.csv (last resort)
    """
    w = WorkspaceClient()
    sources = {}
    for layer, schema in (
        ("deployed_gold",   f"{user_schema_prefix}_gold"),
        ("deployed_silver", f"{user_schema_prefix}_silver"),
        ("deployed_bronze", f"{user_schema_prefix}_bronze"),
    ):
        try:
            tables = list(w.tables.list(catalog_name=catalog, schema_name=schema))
            sources[layer] = {"schema": f"{catalog}.{schema}", "table_count": len(tables)} if tables else None
        except Exception:
            sources[layer] = None
    yaml_dir = Path("gold_layer_design/yaml")
    if yaml_dir.exists() and any(yaml_dir.glob("*.yaml")):
        sources["gold_design"] = {"path": str(yaml_dir), "yaml_count": len(list(yaml_dir.glob("*.yaml")))}
    else:
        sources["gold_design"] = None
    csvs = list(Path("data_product_accelerator/context").glob("*.csv"))
    sources["source_csv"] = {"paths": [str(c) for c in csvs]} if csvs else None
    return sources

Step 0.2 — Select the planning source by mode

Mode	Allowed selected_layer values	Selection rule
acceleration (default)	deployed_gold, gold_design	Pick deployed_gold if present; else gold_design ONLY when explicitly accepted; else STOP with a Gold-required remediation message.
workshop	deployed_gold, gold_design, deployed_silver, deployed_bronze, source_csv	Pick the highest-priority source present. Never silently fall through to a lower layer when a higher one exists.

Acceleration STOP message:

Planning in acceleration mode requires the Gold layer. Run the Gold Layer Design and Setup skills first, or re-run with planning_mode: workshop to plan from a lower layer.

Workshop selection log (must be printed):

Phase 0 — Planning source selected: <selected_layer>
  Available: deployed_gold=<bool>, gold_design=<bool>, deployed_silver=<bool>, deployed_bronze=<bool>, source_csv=<bool>
  Reason: highest-fidelity available input under workshop mode

Step 0.3 — Derive readiness markers

Compute the readiness fields that every emitted manifest must include. requires_gold_promotion is advisory only — it is a hint for production hardening, never a deployment gate.

def readiness_for(selected_layer: str, mode: str) -> dict:
    if selected_layer == "deployed_gold":
        # Production-deployable from Gold.
        return {"implementation_readiness": "gold_ready",
                "requires_gold_promotion": False}
    if selected_layer == "gold_design":
        # Deployable once the Gold layer is provisioned.
        return {"implementation_readiness": "gold_design_only",
                "requires_gold_promotion": False}
    # Silver, Bronze, source CSV — workshop only.
    if mode != "workshop":
        raise SystemExit("Non-Gold planning sources are only allowed in workshop mode.")
    if selected_layer in {"deployed_silver", "deployed_bronze"}:
        # Workshop builds the semantic layer directly on top of Silver/Bronze.
        # Gold promotion is recommended for production but not required to deploy.
        return {"implementation_readiness": "workshop_deployable",
                "requires_gold_promotion": False}
    if selected_layer == "source_csv":
        # No live tables — planning contract only; downstream stages will not
        # attempt to deploy until at least one live layer exists.
        return {"implementation_readiness": "workshop_draft",
                "requires_gold_promotion": False}
    raise SystemExit(f"Unknown selected_layer={selected_layer!r}")

Readiness state semantics:

implementation_readiness	When	Downstream behavior
gold_ready	Acceleration or workshop on deployed_gold	Full production deploy
gold_design_only	Acceleration or workshop on gold_design (Gold YAML, no live tables yet)	Deploy after Gold provisioning; live-catalog checks advisory
workshop_deployable	Workshop on deployed_silver or deployed_bronze	Deploy semantic layer / Genie Spaces directly against the Silver or Bronze schema; Gold promotion is an advisory next step
workshop_draft	Workshop on source_csv only	Planning contract only; downstream stages stop and ask for at least one live layer

Step 0.4 — Stamp planning_source onto every manifest

Every manifest emitted by Phases 1–3 (semantic-layer, observability, ml, genai-agents, gold-dependency, source-dependency) MUST carry a top-level block:

planning_source:
  selected_layer: deployed_gold | gold_design | deployed_silver | deployed_bronze | source_csv
  schema: "<catalog>.<schema>"            # e.g. main.acme_gold (omit/null for source_csv)
  source_yaml_dir: "gold_layer_design/yaml"   # only when selected_layer = gold_design
  source_csv_paths: ["data_product_accelerator/context/<file>.csv"]  # only for source_csv
  selected_at: "<ISO-8601 UTC>"
implementation_readiness: gold_ready | gold_design_only | workshop_deployable | workshop_draft
requires_gold_promotion: true | false   # advisory only; never a deployment gate

Downstream orchestrators (semantic-layer, observability, ml, genai-agents) read these fields:

• gold_ready / gold_design_only / workshop_deployable — proceed with deployment against the layer the manifest declares (gold_schema for Gold sources; silver_schema / bronze_schema for workshop deployments on Silver/Bronze).
• workshop_draft (only emitted when selected_layer = source_csv) — stop before deployment; the plan is a contract only.