dataops

Stars0

Forks0

UpdatedJune 10, 2026 at 12:49

Design, implement, and review DataOps workflows for data platforms. Use for CI/CD for data pipelines, infrastructure as code, deployment automation, monitoring, alerting, rollback plans, cost controls, secrets management, and platform reliability.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

datacoolie

datacoolie/dekit

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Network and Computer Systems AdministratorsComputer and Mathematical Occupations·SOC 15-1244

SKILL.md

readonly

name	dataops
description	Design, implement, and review DataOps workflows for data platforms. Use for CI/CD for data pipelines, infrastructure as code, deployment automation, monitoring, alerting, rollback plans, cost controls, secrets management, and platform reliability.

DataOps

Overview

Build reliable, repeatable, observable data platform operations. Automate what humans forget.

Core Checklist

Before marking DataOps work complete, verify:

Infrastructure is declarative or scriptable and safe to re-run.
Secrets are referenced through vaults, environment injection, or platform secret stores.
CI/CD has build, test, deploy, and rollback gates appropriate to risk.
Rollback or recovery path is explicit and avoids data loss.
Monitoring covers job success/failure, freshness, SLA, cost, and resource utilization.
Alerts are actionable: owner, threshold, impact, and first response are clear.
Cost controls exist: scaling limits, idle shutdown, lifecycle policies, or budgets.
Security baseline covers least privilege, network boundary, encryption, and audit logs.
Runbooks or internal wiki pages are updated when behavior changes.

Patterns

CI/CD For Data

Validate metadata, schemas, SQL, notebooks, and pipeline configs before deploy.
Promote environments explicitly: dev -> staging -> prod.
Require approval for production deploys and breaking data contract changes.
Package artifacts reproducibly: wheels, jars, SQL bundles, notebooks, or deployment manifests.

Infrastructure Provisioning

Prefer IaC or checked-in deployment definitions over manual console changes.
Keep environment parity: prod-like topology, scaled down where needed.
Make provisioning idempotent; repeated runs must converge, not duplicate resources.
Treat destructive changes as irreversible operations requiring explicit approval.

Operations

Define runtime, freshness, and cost baselines for recurring jobs.
Capture structured logs with job id, run id, source, target, row counts, and quality results.
Alert before SLA breach where possible.
Make failed run recovery deterministic: replay window, checkpoint reset, or rollback.

Cost And Capacity

Size compute from observed data volume and SLA.
Use autoscaling carefully; cap runaway scale.
Compact small files and apply lifecycle policies for cold data.
Track cost per pipeline, domain, or product where the platform supports it.

More from this repository

same repository

wiki

datacoolie/dekit

Build and maintain an internal technical LLM wiki when the user requests wiki work or a verified major or architectural change qualifies for automatic wiki review. Use for project memory, architecture notes, contracts, runbooks, decisions, glossary, incremental ingest, delta status, and wiki health. Do not use merely because a routine task or plan completed.

2026-07-180

brainstorm

datacoolie/dekit

Explore ambiguous ideas and solution directions before evidence-backed research or implementation planning. Use for ideation, framing, assumption checks, option generation, product or architecture trade-offs, and deciding whether research or a plan is needed.

2026-07-030

research

datacoolie/dekit

Research technical options with source-backed recommendations. Use for technology evaluation, architecture trade-offs, library/framework choices, best practices, scalability, security, maintainability, or when current external facts matter.

2026-07-030

plan

datacoolie/dekit

Create implementation plans, architecture decisions, and phased roadmaps. Use for Standard or Complex changes, new pipelines, schema changes, migrations, multi-file features, technology choices, or when acceptance criteria and verification strategy are unclear.

2026-06-170

code-review

datacoolie/dekit

Review code and data pipeline changes for correctness, security, performance, maintainability, contract breaks, and missing verification. Use before merge, after implementation, for PRs, commits, pending diffs, or codebase risk scans.

2026-06-100

data-ingestion

datacoolie/dekit

Design data ingestion into landing/bronze layers. Use for source onboarding, transfer pattern selection, full/incremental/CDC scope, change detection, landing zones, schema drift, idempotent file or batch processing, and ingestion quality gates.

2026-06-100

name	dataops
description	Design, implement, and review DataOps workflows for data platforms. Use for CI/CD for data pipelines, infrastructure as code, deployment automation, monitoring, alerting, rollback plans, cost controls, secrets management, and platform reliability.

DataOps

Overview

Build reliable, repeatable, observable data platform operations. Automate what humans forget.

Core Checklist

Before marking DataOps work complete, verify:

Infrastructure is declarative or scriptable and safe to re-run.
Secrets are referenced through vaults, environment injection, or platform secret stores.
CI/CD has build, test, deploy, and rollback gates appropriate to risk.
Rollback or recovery path is explicit and avoids data loss.
Monitoring covers job success/failure, freshness, SLA, cost, and resource utilization.
Alerts are actionable: owner, threshold, impact, and first response are clear.
Cost controls exist: scaling limits, idle shutdown, lifecycle policies, or budgets.
Security baseline covers least privilege, network boundary, encryption, and audit logs.
Runbooks or internal wiki pages are updated when behavior changes.

Patterns

CI/CD For Data

Validate metadata, schemas, SQL, notebooks, and pipeline configs before deploy.
Promote environments explicitly: dev -> staging -> prod.
Require approval for production deploys and breaking data contract changes.
Package artifacts reproducibly: wheels, jars, SQL bundles, notebooks, or deployment manifests.

Infrastructure Provisioning

Prefer IaC or checked-in deployment definitions over manual console changes.
Keep environment parity: prod-like topology, scaled down where needed.
Make provisioning idempotent; repeated runs must converge, not duplicate resources.
Treat destructive changes as irreversible operations requiring explicit approval.

Operations

Define runtime, freshness, and cost baselines for recurring jobs.
Capture structured logs with job id, run id, source, target, row counts, and quality results.
Alert before SLA breach where possible.
Make failed run recovery deterministic: replay window, checkpoint reset, or rollback.

Cost And Capacity

Size compute from observed data volume and SLA.
Use autoscaling carefully; cap runaway scale.
Compact small files and apply lifecycle policies for cold data.
Track cost per pipeline, domain, or product where the platform supports it.