dstack-pipeline-health

name	dstack-pipeline-health
description	Data pipeline health check. Checks data freshness, row count trends, and schema anomalies across tables. Outputs a health dashboard with OK/WARN/FAIL per table. Use for daily pipeline monitoring, SLA checks, or debugging stale data. Trigger phrases: "check pipeline health", "is my data fresh?", "data SLA", "pipeline monitoring", "are my tables up to date?", "something seems stale".
license	MIT
metadata	{"author":"Upsolve Labs"}

Update Check (run first)

_UPD=$(~/.claude/skills/data-stack/bin/data-stack-update-check 2>/dev/null || .claude/skills/data-stack/bin/data-stack-update-check 2>/dev/null || true)
[ -n "$_UPD" ] && echo "$_UPD" || true

If output shows UPGRADE_AVAILABLE <old> <new>: read ~/.claude/skills/data-stack/skills/dstack-upgrade/SKILL.md and follow the "Inline upgrade flow". If JUST_UPGRADED <from> <to>: tell user "Running data-stack v{to} (just updated!)" and continue.

/pipeline-health

You are running a health check across data pipelines or tables.

Before Starting

If the user has specified tables, use those. Otherwise discover them first:

analyze_data("List all available tables and their most recent record timestamp.")

Phase 1: Freshness Check

analyze_data("For these tables: <tables>. Show: (1) most recent record timestamp, (2) how many hours ago that was, (3) row count for today vs 7 days ago.", thread_id=<id>)

Freshness thresholds (infer expected cadence from 7-day history if not specified):

• OK — updated within expected window
• WARN — 1.5× past expected cadence
• FAIL — 2× past expected cadence, or no data today

Phase 2: Row Count Anomalies

analyze_data("For each table, compare today's row count to the 7-day average. Flag any table where today's count is less than 50% or more than 200% of the average.", thread_id=<id>)

Phase 3: Schema Drift

analyze_data("Are there any columns with unusual null rates today compared to last week? Any columns that appear to have stopped receiving data?", thread_id=<id>)

Phase 4: Output Health Dashboard

PIPELINE HEALTH REPORT
As of: <timestamp>
──────────────────────────────────────────────────────────────────
Table              Freshness       Row Δ (7d avg)   Schema    Status
──────────────────────────────────────────────────────────────────
<table_1>          <X>h ago        +3%              OK        ✓ OK
<table_2>          <X>h ago        -62%             —         ⚠ WARN
<table_3>          <X>h ago        no data today    DRIFT     ✗ FAIL

ISSUES:
  FAIL  <table>: <specific issue>
  WARN  <table>: <specific issue>

RECOMMENDED ACTIONS:
  1. <most critical action>
  2. <second action>

Rules

• Run all three phases even if Phase 1 already shows failures.
• If no expected cadence is known, infer it from the last 7 days of update patterns.
• Do not speculate on causes — report observations, recommend investigation.
• If a table has no timestamp column, note it as "freshness unverifiable".