Redefining-DataEngineering-With-AI

Updates an existing Sprint Backlog with new information. Reads the existing backlog, epics, and stories, then merges updated LLD requirements, changed upstream artifacts, revised team capacity, or sprint re-planning. Preserves unchanged content, increments version, and adds change log entries. Use when the user asks to update, revise, or modify existing stories or epics.

Marks Scrum stories Done and rolls epic/backlog status up once every child is complete. BLOCKS completion if any story AC checkbox is unchecked, any dependency is not Done, any referenced deliverable is missing, or any epic-level AC is still open. Uses Edit (not Write) to flip only the Status cell and AC checkboxes, preserving every other line. If the gate blocks on ANY target, NOTHING is edited — the run is atomic from the user's view. Use when the user asks to: - Mark STORY-NN-NNN complete / close a story - Close out EPIC-NN / roll up epic status - "Are we done with sprint 3 yet?" (gate run without edits is acceptable)

Generates an Airflow DAG from the LLD artifact and pipeline configs. Reads the approved LLD from outputs/, applies DAG templates, and writes a production-ready DAG file to airflow/dags/. Also known as: dag generation, pipeline scaffolding, airflow pipeline creation. Input formats: LLD markdown, DAG config YAML. Output format: Python DAG file (.py). Use when the user asks to: - Create, generate, or scaffold an Airflow DAG - Translate an LLD into a runnable pipeline - Build a DAG for bronze/silver/gold layers

Generates layer-scoped local deploy smoke artifacts: shell scripts under _infra/cd/ that re-apply DDL migrations and re-sync the Airflow DAG bag, plus a pytest smoke module under tests/integration/ that drives those scripts and re-triggers the layer DAG end-to-end. Pairs with the scrum-master deploy-validation story type 1:1. Use when the user asks to: - Author the local deploy smoke for a layer (bronze / silver / gold) - Wire up _infra/cd/ scripts (DAG sync, DDL apply) for local - Implement a deploy-validation story (STORY-NN-NNN typed deploy-validation)

Generates the Gold layer of the Patient 360 Medallion pipeline from the approved LLD §2.3 / §5.3, DMS §4, STM Silver-to-Gold sheet, DQS §2-3 (Gold rules), and a given story. Emits PySpark builder modules for the Gold consumer tables declared in LLD §5.3 (currently `patient_summary`, `patient_clinical_history`, `patient_billing_summary` — derived at runtime, not hardcoded), per-table contracts, per-table Spark Expectations DQ rule files, unit tests, and the DAG wiring for the `gold_build` task group. Modes: - Story mode: STORY-NN-NNN — emit only Gold deliverables in scope - Full mode: process every un-Done story classified as `gold`, topo-sorted by Depends On, until the Gold layer is complete Project-agnostic: the table list, source-table joins, business-invariant DQ rules, and module names are ALL read from the LLD §5.3, DMS §4, STM Silver-to-Gold sheet, and DQS §2-3 at runtime. Use when the user asks to: - Generate or update Gold layer code for a story or epic - Build the Gold consumer tables (patient_sum

Generates the Bronze config-driven ingestion layer from the approved LLD, DMS, STM, DQS, and a given story. Modes: - Story mode: read a STORY-NN-NNN, extract backtick-quoted deliverables from its Acceptance Criteria, generate only those that fall under this skill's domain of ownership. ROUTE-OUT for others. - Full mode: process every un-Done story classified as `ingestion`, topo-sorted by Depends On. Project-agnostic: the table list, critical-table designation, per-table DQ rule filenames, and runner module names are ALL read from the current run's LLD / DMS / STM / DQS / stories at runtime. No project identifiers or story IDs are hardcoded. Use when the user asks to: - Generate or update Bronze ingestion code for a story - Build the ingestion runner / factory / SparkSubmit wrapper - Generate per-table ingestion configs

Generates CI/CD pipeline configuration (GitHub Actions / GitLab CI) from the project structure and LLD artifact. Produces lint, test, and deploy workflow files targeting the medallion pipeline. Use when the user asks to: - Create, generate, or scaffold a CI/CD pipeline - Set up GitHub Actions or GitLab CI for the project - Automate testing and deployment of the data pipeline

Generates the foundation layer of a cookiecutter-chapter project: directory tree via cookiecutter render, pyproject.toml, Makefile, docker-compose, cross-cutting utility modules under `src/<project>/utils/`, StructType schema contracts, and test-harness scaffolding. Foundation assets that every other generator (create-dag, create-ingestion, create-pipeline) assumes exist. Project-agnostic: all project-specific facts (table lists, module names, interface contracts) are read from the current run's LLD / DMS / story file at runtime. Use when the user asks to: - Scaffold the project / bootstrap the chapter - Implement a foundation-layer story (utility modules, config loader, contracts, docker-compose)

Generates the Silver layer of the Patient 360 Medallion pipeline from the approved LLD, DMS, STM, DQS, and a given story or epic. Emits per-table PySpark transformation modules (SCD2 dimensions + cleansed facts), the shared SCD2 helper, per-table contracts, per-table Spark Expectations DQ rule files, unit tests, and the DAG wiring for the `silver_dimensions` and `silver_facts` task groups. Modes: - Story mode: read a STORY-NN-NNN, extract backtick-quoted deliverables from its Acceptance Criteria, generate only those that fall under the Silver layer (modules under `src/patient_360/silver/`, contracts/dq files for silver-prefixed tables, tests under `tests/silver/`). ROUTE-OUT any deliverable that lives in another layer. - Full mode: process every un-Done story classified as `silver`, topo-sorted by Depends On, until the Silver layer is complete. Project-agnostic: the table list, SCD2 dimension designation, hash column list, per-table DQ rule filenames, and module names are ALL read from the current

Drives the implementation of one or more Scrum stories by dispatching to the correct downstream create-/update- skill per epic. Accepts a story ID, a comma-separated list, an epic ID (topo-sorted by Depends On), or a Sprint label (e.g. "Sprint 3"). Never edits story markdown itself — it only orchestrates the generators and surfaces their results. Use when the user asks to: - Implement STORY-NN-NNN or a list of stories - "Implement EPIC-02" / "Implement all of Sprint 3" - Build the next story in the backlog - Drive story-mode generation across several stories in one turn

Re-verifies every row in `inputs/code/v*/LIBRARIES.md` against upstream sources (context7, PyPI JSON, GitHub releases), prints a diff, asks the user for approval via AskUserQuestion, and rewrites `LIBRARIES.md` with a fresh `last_verified` timestamp. Invoke standalone (`/developer-plugin:refresh-libraries`) or automatically from any create-*/update-* skill when the cached versions are stale. Use when the user asks to: - Refresh / bump / update library versions in the pattern handbook - Check if pinned libraries are current - Respond "Refresh now" to a staleness prompt from another skill

Updates an existing Airflow DAG to reflect changes in the LLD or config. Applies incremental edits, bumps version comment, and re-validates. Use when the user asks to: - Update, modify, or patch an existing DAG - Reflect an LLD change in the pipeline - Add or remove tasks from an existing DAG

Updates an existing Gold layer implementation in response to changes in the upstream LLD §5.3, DMS §4, STM Silver-to-Gold sheet, DQS §2-3, or in the Silver layer that Gold reads from. Preserves unchanged builders, increments versioning per the 3-scenario rule, and warns when Silver inputs have changed in a way that requires a Gold rebuild. Modes: - Story mode: STORY-NN-NNN — apply only story-authorized changes - Diff mode (default): compare current Gold against the latest upstream + Silver contracts and update any drifted builder - Full mode: regenerate every Gold builder Use when the user asks to: - Update or revise Gold layer code after a Silver / DMS / STM / DQS change - Reconcile Gold builders with re-approved upstream artifacts - Apply ripple changes from `update-silver` to Gold

Updates the Bronze ingestion framework (runner, factory, spark-submit wrapper, and per-table YAML configs) to reflect changes in the latest LLD or STM. Applies minimal incremental edits — never rewrites unchanged files. Use when the user asks to: - Update, modify, or patch the ingestion code - Add, remove, or rename a Bronze table from configs - Reflect an LLD or STM change in the ingestion configs - Change empty-input behavior, retries, timeout, or DQ action for a table

Updates an existing project scaffold: adds missing directories/modules, patches pyproject.toml / Makefile / docker-compose when the LLD changes, and brings StructType schema contracts in sync with a revised DMS. Never deletes files. Use when the user asks to: - Refresh scaffold after LLD revision - Sync contracts/ against a new DMS version - Add a new foundation module to an existing project

Updates an existing Silver layer implementation in response to changes in the upstream LLD §5.2, DMS §3, STM Bronze-to-Silver sheet, or DQS §2. Preserves unchanged modules, increments versioning per the 3-scenario rule (defined in this skill's Phase 2 table), and surfaces ripple effects to the Gold layer. Modes: - Story mode: STORY-NN-NNN — apply only the changes the story authorizes - Diff mode (default): compare current Silver implementation against the latest approved LLD/DMS/STM/DQS and update any drifted module - Full mode: regenerate every Silver module against the latest upstream Use when the user asks to: - Update or revise Silver layer code after a DMS / STM / DQS change - Apply LLD edits to existing Silver transforms - Reconcile Silver code with a re-approved upstream artifact set

Validates an Airflow DAG file for correctness, import errors, and compliance with project conventions. Runs static checks and reports CRITICAL / WARNING / INFO findings. Use when the user asks to: - Validate, check, or lint a DAG - Verify a DAG has no import errors - Confirm DAG follows project standards

Validates an existing Gold layer implementation against the latest approved LLD §5.3, DMS §4, STM Silver-to-Gold sheet, and DQS §2-3 Gold rules. Produces a severity-ranked report covering builder presence, schema alignment, SCD2 `is_current` filtering, DQ-before-write order, contract/DQ coverage, DAG wiring, and traceability. Use when the user asks to: - Validate, check, review, or audit the Gold layer - Run quality checks on Gold builders before promotion - Find drift between Gold code and Silver / DMS / STM / DQS

Validates the Bronze ingestion framework for correctness against LLD §2.3 and §5.1. Runs static checks on the ingestion runner, factory, SparkSubmit wrapper, and per-table YAML configs. Reports findings as CRITICAL / WARNING / INFO. Use when the user asks to: - Validate, check, or lint the ingestion code - Verify every Bronze table has a config file - Confirm per-table configs match the LLD empty-input / retry / DQ spec

Validates the project scaffold against the LLD (directory tree, Make targets, infra layout) and the DMS (StructType contracts). Read-only: reports pass/fail per check without editing anything. Runs the scaffold's own smoke tests (`uv sync`, import check, `pytest --collect-only`) and surfaces each failure with the exact command to reproduce. Use when the user asks to: - Validate the scaffold / foundation - Check a foundation-layer story's implementation - Confirm scaffold is ready before running create-dag / create-ingestion

Validates an existing Silver layer implementation against the latest approved LLD §5.2, DMS §3, STM Bronze-to-Silver sheet, and DQS §2. Produces a severity-ranked report (CRITICAL / WARNING / INFO) covering module presence, schema alignment, SCD2 wiring, DQ-before-write order, contract/DQ coverage, and traceability. Use when the user asks to: - Validate, check, review, or audit the Silver layer - Run quality checks on Silver code before promotion - Find drift between Silver code and upstream artifacts

Verifies that the code implementation satisfies every acceptance criterion of one or more Scrum stories (and every epic-level AC for an epic target). Combines a static-heuristic AC scan (keyword/path checks derived from the create-*/update-* skills' Story → Deliverable Map) with the matching downstream validate-* skill (validate-ingestion / validate-dag / validate-pipeline). Read-only — never flips Status or AC checkboxes. Use when the user asks to: - Validate STORY-NN-NNN / EPIC-NN against its acceptance criteria - Check AC compliance before marking a story Done - Confirm an implementation is ready to close out

Converts a Data Quality Specification (DQS) markdown document into per-table Spark-Expectations YAML rule files. Reads the DQS, groups rules by target table, and generates one YAML file per table compatible with spark-expectations >= 2.6.0. Also known as: SE rules generation, Spark-Expectations config, DQS-to-YAML conversion, quality rules export. Input formats: DQS Markdown (.md) file. Output format: Per-table YAML files in outputs/dqs/v{N}/se-rules/. Use when the user asks to: - Generate SE rules or Spark-Expectations YAML - Convert DQS to YAML rule files - Produce Spark-Expectations configs from a DQS - Export quality rules for spark-expectations framework - "Create the SE YAML files from the DQS"

Generates a Sprint Backlog (Epics and Stories) from the LLD and all upstream artifacts. Reads the latest LLD, DRD, HLD, DMS, STM, DQS, and scrum master inputs. Produces a structured backlog with individually filed Epics and Stories traceable to upstream artifacts. Also known as: story writing, work decomposition, sprint planning, backlog creation. Input formats: LLD Markdown + all upstream artifacts + team capacity document. Output format: BACKLOG index (.md) + individual EPIC and STORY Markdown files. Use when the user asks to: - Create, generate, draft, or write stories from the LLD - Decompose the LLD into epics and stories - Build a sprint backlog or sprint plan - Break down the implementation plan into work items - Start story writing from the latest design documents

Validates a Low-Level Design (LLD) document against completeness and quality standards. Checks all 14 required sections, upstream artifact references, DAG specification, code architecture, configuration schema, and traceability. Reports issues as CRITICAL, WARNING, or INFO with suggested fixes. Also known as: LLD review, implementation design audit, tech spec validation. Input formats: LLD Markdown (.md) file. Output format: Validation report with severity-ranked findings. Use when the user asks to: - Validate, check, review, verify, or audit an LLD - Assess LLD completeness or implementation quality - Find issues or gaps in a low-level design document - Run quality checks on an LLD before development - "Check if the LLD is ready for the development team"

Generates a Data Requirements Document (DRD) from business inputs. Reads business requests, stakeholder interviews, source system docs, and data catalogs from an input folder. Produces a business-friendly DRD following a standard Jinja2 template. Also known as: data requirements gathering, requirements analysis, business-to-data translation, requirements elicitation. Input formats: business requests (.md), stakeholder interviews (.md), source system docs (.md), data catalogs (.md). Output format: Markdown (.md) DRD document. Use when the user asks to: - Create, generate, draft, or write a DRD - Analyze business documents into data requirements - Gather or elicit data requirements from inputs - Translate a business request into structured requirements - "What data do we need for this project?" - Start a new data requirements analysis

Converts a Data Quality Specification (DQS) markdown document into per-table Spark-Expectations YAML rule files. Reads the DQS, groups rules by target table, and generates one YAML file per table compatible with spark-expectations >= 2.6.0. Also known as: SE rules generation, Spark-Expectations config, DQS-to-YAML conversion, quality rules export. Input formats: DQS Markdown (.md) file. Output format: Per-table YAML files in outputs/dqs/v{N}/se-rules/. Use when the user asks to: - Generate SE rules or Spark-Expectations YAML - Convert DQS to YAML rule files - Produce Spark-Expectations configs from a DQS - Export quality rules for spark-expectations framework - "Create the SE YAML files from the DQS"

Generates a Low-Level Design (LLD) document from all upstream artifacts (DRD, HLD, DMS, STM, DQS) and technical lead inputs. Reads development standards, infrastructure specs, and orchestration patterns. Produces a structured LLD covering DAG specification, code architecture, file formats, performance optimization, configuration schema, and deployment procedures. Also known as: low-level design, implementation design, technical design, engineering design, build specification. Input formats: DRD + HLD + DMS + STM + DQS Markdown/Excel + technical inputs. Output format: Markdown (.md) LLD document with 14 sections. Use when the user asks to: - Create, generate, draft, or write an LLD - Design the implementation for the pipeline - Translate architecture into a build-ready specification - Start a new LLD from upstream artifacts - "Create the low-level design for the project"

Extracts configuration parameters from LLD Section 7 (Configuration Schema) and generates environment-specific YAML configuration templates. Produces one YAML per environment (`dev.yaml`, `stage.yaml`, `prod.yaml`) plus a combined `config-template.yaml`, each with a top-level `scaffold_target:` pointing at the cookiecutter scaffold path (`_infra/cd/config/<env>.yaml`) where the developer drops the file in chapter-5. Also known as: config generation, environment config, YAML config export, configuration template, deployment config. Input formats: LLD Markdown (.md) file with Section 7 parameter table. Output format: YAML config-template.yaml in outputs/lld/v{N}/config/. Use when the user asks to: - Generate configuration templates from an LLD - Create environment-specific config YAML - Export config parameters from the LLD - Produce deployment configuration files - "Create the config template from the LLD"

Updates an existing Low-Level Design document with new information. Reads the existing LLD and merges updated upstream artifacts, infrastructure changes, DAG revisions, or configuration updates. Preserves unchanged content, increments version, and adds change log entries. Also known as: LLD revision, implementation design update, tech spec update. Input formats: Existing LLD (.md) + changed upstream artifacts or inputs. Output format: Updated Markdown (.md) LLD document. Use when the user asks to: - Update, revise, modify, or amend an existing LLD - Incorporate new infrastructure specs or orchestration patterns - Adjust DAG configuration or deployment settings - Apply changes from updated upstream artifacts - "Update the LLD with the new Spark cluster sizing"

Validates a Low-Level Design (LLD) document against completeness and quality standards. Checks all 14 required sections, upstream artifact references, DAG specification, code architecture, configuration schema, and traceability. Reports issues as CRITICAL, WARNING, or INFO with suggested fixes. Also known as: LLD review, implementation design audit, tech spec validation. Input formats: LLD Markdown (.md) file. Output format: Validation report with severity-ranked findings. Use when the user asks to: - Validate, check, review, verify, or audit an LLD - Assess LLD completeness or implementation quality - Find issues or gaps in a low-level design document - Run quality checks on an LLD before development - "Check if the LLD is ready for the development team"

Reviews pending corrections from the developer learnings queue and applies them as generalized rules to the relevant skill's Learnings & Corrections section. Uses the Reflect-Abstract-Generalize-Write pattern to convert raw user corrections into absolute directives that improve future skill execution. Use when the user asks to: - Apply learnings or corrections - Review the learnings queue - Improve developer skills from past feedback - "What corrections have accumulated?"

Marks Scrum stories Done and rolls epic/backlog status up once every child is complete. BLOCKS completion if any story AC checkbox is unchecked, any dependency is not Done, any referenced deliverable is missing, or any epic-level AC is still open. Uses Edit (not Write) to flip only the Status cell and AC checkboxes, preserving every other line. If the gate blocks on ANY target, NOTHING is edited — the run is atomic from the user's view. Use when the user asks to: - Mark STORY-NN-NNN complete / close a story - Close out EPIC-NN / roll up epic status - "Are we done with sprint 3 yet?" (gate run without edits is acceptable)

Generates an Airflow DAG from the LLD artifact and pipeline configs. Reads the approved LLD from outputs/, applies DAG templates, and writes a production-ready DAG file to airflow/dags/. Also known as: dag generation, pipeline scaffolding, airflow pipeline creation. Input formats: LLD markdown, DAG config YAML. Output format: Python DAG file (.py). Use when the user asks to: - Create, generate, or scaffold an Airflow DAG - Translate an LLD into a runnable pipeline - Build a DAG for bronze/silver/gold layers

Generates layer-scoped local deploy smoke artifacts: shell scripts under _infra/cd/ that re-apply DDL migrations and re-sync the Airflow DAG bag, plus a pytest smoke module under tests/integration/ that drives those scripts and re-triggers the layer DAG end-to-end. Pairs with the scrum-master deploy-validation story type 1:1. Use when the user asks to: - Author the local deploy smoke for a layer (bronze / silver / gold) - Wire up _infra/cd/ scripts (DAG sync, liquibase apply) for local - Implement a deploy-validation story (STORY-NN-NNN typed deploy-validation)

Generates the Bronze config-driven ingestion layer from the approved LLD, DMS, STM, DQS, and a given story. Modes: - Story mode: read a STORY-NN-NNN, extract backtick-quoted deliverables from its Acceptance Criteria, generate only those that fall under this skill's domain of ownership. ROUTE-OUT for others. - Full mode: process every un-Done story classified as `ingestion`, topo-sorted by Depends On. Project-agnostic: the table list, critical-table designation, per-table DQ rule filenames, and runner module names are ALL read from the current run's LLD / DMS / STM / DQS / stories at runtime. No project identifiers or story IDs are hardcoded. Use when the user asks to: - Generate or update Bronze ingestion code for a story - Build the ingestion runner / factory / SparkSubmit wrapper - Generate per-table ingestion configs

Generates layer-scoped integration tests under tests/integration/ that trigger an Airflow DAG against the local docker-compose stack (Airflow + Unity Catalog OSS + Marquez) and assert end-to-end data quality. Reads the approved LLD and the target story's ACs, emits a per-layer test module (DAG trigger + UC table checks) plus a SE-runtime-evidence module (bronze_se_stats / Marquez dq_pass_rate). Pairs with the scrum-master integration-test story type 1:1. Use when the user asks to: - Author the integration test for a layer (bronze / silver / gold) - Wire up a pytest-mark integration suite against the docker stack - Implement an integration-test story (STORY-NN-NNN typed integration-test)

Generates CI/CD pipeline configuration (GitHub Actions / GitLab CI) from the project structure and LLD artifact. Produces lint, test, and deploy workflow files targeting the medallion pipeline. Use when the user asks to: - Create, generate, or scaffold a CI/CD pipeline - Set up GitHub Actions or GitLab CI for the project - Automate testing and deployment of the data pipeline

Generates the foundation layer of a cookiecutter-chapter project: directory tree via cookiecutter render, pyproject.toml, Makefile, docker-compose, cross-cutting utility modules under `src/<project>/utils/`, StructType schema contracts, and test-harness scaffolding. Foundation assets that every other generator (create-dag, create-ingestion, create-pipeline) assumes exist. Project-agnostic: all project-specific facts (table lists, module names, interface contracts) are read from the current run's LLD / DMS / story file at runtime. Use when the user asks to: - Scaffold the project / bootstrap the chapter - Implement a foundation-layer story (utility modules, config loader, contracts, docker-compose)

Drives the implementation of one or more Scrum stories by dispatching to the correct downstream create-/update- skill per epic. Accepts a story ID, a comma-separated list, an epic ID (topo-sorted by Depends On), or a Sprint label (e.g. "Sprint 3"). Never edits story markdown itself — it only orchestrates the generators and surfaces their results. Use when the user asks to: - Implement STORY-NN-NNN or a list of stories - "Implement EPIC-02" / "Implement all of Sprint 3" - Build the next story in the backlog - Drive story-mode generation across several stories in one turn