saptak-add-pack

name	saptak-add-pack
version	0.2.0
description	Add a new pack to Saptak. Walks every required touchpoint in docs/PACK_CONTRACT.md — pack directory scaffolding, compose services (seeder + dbt-runner + rag-seeder + graph-populator + streamer), OPA action prefix, six `*_PACK_*` dict entries in layer_details.py, queries.py SQL constants, workflows.py workflow + activities, main.py endpoints + worker registration, L2 broadcast + SSE endpoint, frontend PACKS + PACK_L7 + SVG dicts + PACK_L2_FEED + ROW_RENDERERS, style.css theme, acceptance test registration. Verifies via boot validator + pack_contract_completeness.py + pack_modality_completeness.py + the full make verify suite. Use when: "add a pack", "create a new pack", "scaffold pack <name>".
allowed-tools	["Read","Write","Edit","Bash","Grep","Glob"]

Saptak — add a pack

The Saptak architecture is intentionally pack-extensible. The infrastructure is pack-agnostic; a new pack is the data + per-pack configuration. There are ~36 touchpoints (was 24 before the 2026-05-22 modality-audit closure added L2 broadcast wiring, hypertable derivation, and the rag-seeder + graph-populator + streamer pattern). Governed by docs/PACK_CONTRACT.md + docs/PACK_MODALITY_AUDIT_2026-05-22.md. This skill walks every one in order.

Critical norm — no shortcuts: every pack must ship every applicable L1 modality (relational, pgvector, AGE graph, TimescaleDB hypertable) AND an L2 streamer. Opt-outs require an explicit modality_opt_outs: waiver block in the pack's manifest.yaml with reviewer-grade rationale (≥25 words explaining what was tried, not deferred). tests/pack_modality_completeness.py enforces this mechanically; CI fails on any silent opt-out.

Before starting

1. Read docs/PACK_CONTRACT.md — the enforceable contract.
2. Read packs/usgs/seed.py — the canonical reference example for "Phase 2a + 3a baked in from day one."
3. Pick names once. Per docs/HOWTO_ADD_A_PACK.md §0, decide:
   • Pack id (lowercase, hyphens): <pack-id>
   • Pack badge (3-4 char uppercase): <PACK>
   • Pack display label: <Pack Label>
   • OPA action prefix: query.<pack-id-alias> (alias = first segment before -)
   • AGE graph name (if shipping graph): saptak_<short>
   • Workflow class name: <Pack>EventNarrationWorkflow or similar

Workflow

Use Explore agent first to survey the EXACT current state of each touchpoint (per the article's "split exploration from editing" pattern). Don't grep-and-edit.

Agent(subagent_type: "Explore",
      prompt: "Survey the Saptak codebase pack-contract surface. For each
               of the 24 touchpoints in docs/PACK_CONTRACT.md, report the
               current state for packs f1, sec-edgar, usgs. Identify the
               file + line range where I'll insert the new pack's entry.
               Do not edit anything.")

Use the survey output to drive each edit. After each edit, verify with the contract:

python3 tests/pack_contract_completeness.py   # static — no Docker needed

When all 24 touchpoints land:

make demo-lite               # boot validator must pass
make verify                  # full acceptance suite must pass
make verify-clean-rebuild    # from-empty proof

The ~36 touchpoints, in order

(Same list as in packs/AGENTS.md — that file is the working copy; this skill is the activation trigger.)

The numbering below preserves the original 24 (steps 1-24) and adds the modality-closure touchpoints (steps 25-36). The closure work landed in PR #6 (2026-05-22); any pack added since must include all 36.

Each touchpoint with file path, what to add, and verification:

Pack directory (1-8)

1. packs/<pack-id>/manifest.yaml — copy + adapt packs/usgs/manifest.yaml
2. packs/<pack-id>/seed.py — copy + adapt packs/usgs/seed.py (reference example). Every CREATE TABLE must include tenant_id + classification. Post-seed: duplicate-for-acme + SELECT enable_tenant_isolation('<table>').
3. packs/<pack-id>/Dockerfile — copy packs/usgs/Dockerfile
4. packs/<pack-id>/dbt/dbt_project.yml — name: saptak_<pack-id>, profile: saptak_<pack-id>, +post-hook: ALTER VIEW {{ this }} SET (security_invoker = true)
5. packs/<pack-id>/dbt/profiles.yml — copy + adapt packs/usgs/dbt/profiles.yml
6. packs/<pack-id>/dbt/Dockerfile — copy packs/usgs/dbt/Dockerfile
7. packs/<pack-id>/dbt/models/sources.yml — declare the pack's tables as sources
8. packs/<pack-id>/dbt/models/<view>.sql — at least one semantic view

Compose (9-10)

9. compose/docker-compose.yml — add <pack-id>-seeder + <pack-id>-dbt-runner services
10. compose/docker-compose.yml — add to saptak-demo.depends_on

OPA (11)

11. compose/opa/policy.rego — add "query.<pack-id-alias>" to known_action_prefixes

App backend layer config (12-17)

Per docs/PACK_CONTRACT.md §E. The boot validator enforces these.

12. apps/saptak-demo/saptak_demo/layer_details.py — L1_PACK_CONFIG[<alias>]
13. apps/saptak-demo/saptak_demo/layer_details.py — L2_PACK_CONFIG[<alias>]
14. apps/saptak-demo/saptak_demo/layer_details.py — L3_PACK_CONFIG[<alias>]
15. apps/saptak-demo/saptak_demo/layer_details.py — L4_PACK_CONFIG[<alias>]
16. apps/saptak-demo/saptak_demo/layer_details.py — L5_PACK_ACTIONS[<alias>]
17. apps/saptak-demo/saptak_demo/layer_details.py — L7_PACK_WORKFLOWS[<alias>]

App backend code (18-20)

18. apps/saptak-demo/saptak_demo/queries.py — SQL constants (e.g., <PACK>_<NAME>)
19. apps/saptak-demo/saptak_demo/workflows.py — <Pack>EventNarrationWorkflow + 3 activities
20. apps/saptak-demo/saptak_demo/main.py — 3 endpoints (sql, narrated, via-temporal) + worker registration

Frontend (21-22)

21. apps/saptak-demo/saptak_demo/static/app.js — PACKS entry, PACK_L7 entry, SVG_L2_PACK entry, SVG_L3_PACK entry, FLOW_PATHS_BY_RENDERER entries, two renderer functions
22. apps/saptak-demo/saptak_demo/static/style.css — body[data-pack="<alias>"] accent + .pack-case-btn.pack-<alias> theme

Tests (23-24)

23. tests/pack_author_acceptance.py — entry in KNOWN_PACKS (include the new graph + L2 endpoints in endpoints_substr and the new tables in tables_substr)
24. tests/usecase_acceptance.py — <ALIAS>_PACK_CASES list + section_2_X_<pack> runner + register in main(). Include at minimum: SQL killer, RAG-narrated, Temporal-via, AGE-graph cells.

Modality-closure touchpoints (25-36) — added 2026-05-22

The first 24 touchpoints predate the modality audit. These twelve are required for any pack added since 2026-05-22. The lint tests/pack_modality_completeness.py mechanically enforces them.

25. packs/<pack-id>/streamer/streamer.py + Dockerfile — L2 Kafka publisher (real-source poll OR synthesize-source-side, depending on whether a free public real-time feed exists)
26. packs/<pack-id>/rag-seeder/rag_seed.py + Dockerfile — bge-m3 embedder over a domain-specific corpus
27. packs/<pack-id>/graph-populator/populate.py + Dockerfile — Cypher MERGE with tenant_id in every MATCH literal
28. packs/<pack-id>/seed.py — synthesize the corpus the rag-seeder embeds AND the entities the graph-populator builds (single source of truth across both modalities)
29. packs/<pack-id>/seed.py — HYPERTABLE_DERIVATIONS block if any time-series data lacks a typed time column (see packs/f1/seed.py)
30. compose/docker-compose.yml — three new services (rag-seeder, graph-populator, streamer) + add each to saptak-demo.depends_on
31. apps/saptak-demo/saptak_demo/streaming_demo.py — make_<pack>_<thing>_broadcast() + _normalize_<pack>_<thing>() + <pack>_<thing>_event_stream()
32. apps/saptak-demo/saptak_demo/main.py lifespan — app.state.<pack>_<thing>_broadcast = make_…(); start + stop in startup / shutdown
33. apps/saptak-demo/saptak_demo/main.py — new endpoints: /api/stream/<pack-id>-<thing> (SSE bridge), /api/query/<pack-id>-graph-<traversal> (graph killer)
34. apps/saptak-demo/saptak_demo/static/app.js PACK_L2_FEED[<alias>] — has_live_stream: true + row_renderer: "<alias>", ROW_RENDERERS entry, render<Pack><Thing>Item function
35. apps/saptak-demo/saptak_demo/static/app.js PACKS[<alias>].cases — add the graph killer case alongside existing SQL/RAG/Temporal cases; FLOW_PATHS_BY_RENDERER entry; PACK_L7 already exists from step 17
36. packs/<pack-id>/manifest.yaml — IF you legitimately cannot ship a modality, declare modality_opt_outs.<modality>.rationale: block (≥25 words explaining what was tried, not deferred)

Verification (after all 36 land)

# Step 1: static contract check (no Docker)
python3 tests/pack_contract_completeness.py     # 270+ cells; +6-10 per new pack

# Step 1b: modality lint (no Docker) — added 2026-05-22
python3 tests/pack_modality_completeness.py     # 5 modality cells per pack + waiver quality

# Step 2: boot validator (rebuild + restart saptak-demo)
cd compose && docker compose build saptak-demo && \
  docker compose up -d --no-deps saptak-demo

# Step 3: full acceptance suite
make verify                                     # 648+ cells across 5 suites

# Step 4 (reviewer-grade): clean rebuild from empty state
make verify-clean-rebuild                       # must reproduce 648+

If any step fails, the failure message names the missing entry. Fix + re-run.

The modality lint is the gate — if it fails, the pack is incomplete by the post-2026-05-22 norm. Either build the modality or add a modality_opt_outs: waiver with a reviewer-grade rationale paragraph. There's no third option.

Pitfalls — read before you start

These have all bitten previous pack additions. The skill is here partly to make them avoidable.

• The seeder must add tenant_id + classification at CREATE TABLE time (not via post-seed ALTER). Otherwise a clean rebuild doesn't reproduce the tenant-isolation state.
• Duplicate rows for the acme tenant. After loading demo rows: INSERT INTO <table> (..., tenant_id) SELECT ..., 'acme' FROM <table> WHERE tenant_id='demo'. Without this, cross-tenant acceptance tests fail because alice has no data.
• dbt views need security_invoker = true via +post-hook in dbt_project.yml. Otherwise RLS doesn't apply to view queries.
• OPA must be restarted after editing policy.rego. docker compose restart opa.
• The boot validator runs at saptak-demo startup. A broken contract → the app refuses to boot with a clear message naming the missing entry. Don't try to override (SAPTAK_PACK_CONTRACT=skip) — fix the entry.
• Don't grep-and-edit. Use the Explore agent first to enumerate every touchpoint's current state. Pattern-matching on text misses the 6 *_PACK_* dicts that share a naming convention but aren't called out as such.
• No "v0.3 deferred" framing. If you find yourself writing "v0.3 follow-on" or "deferred to v0.3" in a manifest's phase_status.later: block for a modality (pgvector / AGE / TimescaleDB / L2 streamer), STOP — that's the path-of-least-resistance failure mode the modality audit closed. Either build it or write a real waiver under modality_opt_outs: (≥25 words on what was tried).
• No silent opt-outs in L1_PACK_CONFIG. Setting vector_table: None or age_graph: None without a corresponding modality_opt_outs: waiver in the pack manifest fails pack_modality_completeness.py. Build or waive — don't quietly skip.

What "done" means

• python3 tests/pack_contract_completeness.py passes (270+ cells, +6-10 per new pack)
• python3 tests/pack_modality_completeness.py passes (5 modality cells per pack + waiver quality)
• make verify passes (648+ cells across 5 suites)
• make verify-clean-rebuild passes (must reproduce all of the above from empty state)
• The UI shows the new pack's tab with its accent color, all 4 use-case tiles work (SQL + RAG + Graph + Temporal), the L1-L7 drill panels show pack-specific content (not F1 fall-back), and the L2 panel shows the live SSE stream (not a "deferred" placeholder)
• One commit per logical group of touchpoints (pack-directory + compose + opa + backend + frontend + tests + modality-closures = 6-7 commits typically)

Related

• docs/PACK_CONTRACT.md — enforceable contract
• docs/HOWTO_ADD_A_PACK.md — narrative walkthrough
• docs/PACK_ANATOMY.md — cross-pack patterns
• packs/AGENTS.md — pack-author conventions (tool-agnostic, AGENTS.md format)
• packs/usgs/ — canonical reference pack