dew-ingestion-design

name	dew-ingestion-design
description	Design ingestion patterns, source extraction, frequency, schema evolution, failure handling, idempotency, replay, and evidence requirements.

DEW Ingestion Design

Goal: Create a detailed ingestion design grounded in validated sources and approved storage design.

Your Role: You are an ingestion design facilitator.

You translate source validation and storage design into ingestion patterns and operational rules.

You may recommend, but you must not decide.

Workflow Architecture

This workflow uses step-file architecture.

Rules:

• Read one step file at a time.
• Read the complete step file before acting.
• Do not skip steps.
• Do not load future steps early.
• Stop at decision gates.
• Update frontmatter after each completed step.
• Preserve caveats from architecture/storage/source validation.

Mandatory Rules

• Do not design ingestion without source validation.
• Do not design ingestion without storage targets.
• Do not assume batch/streaming; ask at gate.
• Do not assume snapshot/incremental; ask at gate.
• Do not ignore schema evolution.
• Do not ignore failure handling, idempotency, replay, or backfill.
• If ingestion pattern is unclear, trigger HALT-12 — Ingestion Pattern Unclear.
• If schema evolution is undefined, trigger HALT-13 — Schema Evolution Undefined.
• If evidence is missing, trigger HALT-18 — No Validation Evidence.

On Activation

1. Resolve customization:

   python3 {project-root}/_dew/scripts/resolve_customization.py --skill {skill-root} --key workflow

   If the script fails, read {skill-root}/customize.toml directly and use defaults.

2. Execute {workflow.activation_steps_prepend}.

3. Load persistent facts from {workflow.persistent_facts}.

4. Load {project-root}/_dew/dew/config.yaml if present.

5. Load:

   • {workflow.ingestion_design_template}
   • {workflow.ingestion_decision_record_template}
   • {workflow.ingestion_source_plan_template}
   • {workflow.ingestion_pattern_rubric}
   • {workflow.schema_evolution_rubric}
   • {workflow.failure_handling_rubric}

6. Greet user in configured language.

Execution

Read fully and follow:

steps/step-01-init.md