| name | dataops |
| description | Design, implement, and review DataOps workflows for data platforms. Use for CI/CD for data pipelines, infrastructure as code, deployment automation, monitoring, alerting, rollback plans, cost controls, secrets management, and platform reliability. |
DataOps
Overview
Build reliable, repeatable, observable data platform operations. Automate what humans forget.
Core Checklist
Before marking DataOps work complete, verify:
- Infrastructure is declarative or scriptable and safe to re-run.
- Secrets are referenced through vaults, environment injection, or platform secret stores.
- CI/CD has build, test, deploy, and rollback gates appropriate to risk.
- Rollback or recovery path is explicit and avoids data loss.
- Monitoring covers job success/failure, freshness, SLA, cost, and resource utilization.
- Alerts are actionable: owner, threshold, impact, and first response are clear.
- Cost controls exist: scaling limits, idle shutdown, lifecycle policies, or budgets.
- Security baseline covers least privilege, network boundary, encryption, and audit logs.
- Runbooks or internal wiki pages are updated when behavior changes.
Patterns
CI/CD For Data
- Validate metadata, schemas, SQL, notebooks, and pipeline configs before deploy.
- Promote environments explicitly: dev -> staging -> prod.
- Require approval for production deploys and breaking data contract changes.
- Package artifacts reproducibly: wheels, jars, SQL bundles, notebooks, or deployment manifests.
Infrastructure Provisioning
- Prefer IaC or checked-in deployment definitions over manual console changes.
- Keep environment parity: prod-like topology, scaled down where needed.
- Make provisioning idempotent; repeated runs must converge, not duplicate resources.
- Treat destructive changes as irreversible operations requiring explicit approval.
Operations
- Define runtime, freshness, and cost baselines for recurring jobs.
- Capture structured logs with job id, run id, source, target, row counts, and quality results.
- Alert before SLA breach where possible.
- Make failed run recovery deterministic: replay window, checkpoint reset, or rollback.
Cost And Capacity
- Size compute from observed data volume and SLA.
- Use autoscaling carefully; cap runaway scale.
- Compact small files and apply lifecycle policies for cold data.
- Track cost per pipeline, domain, or product where the platform supports it.
