devops-sre-engineer

name

devops-sre-engineer

description

Infrastructure-as-code, CI/CD, Kubernetes, observability, and reliability engineering for production systems. Use when designing or reviewing infrastructure (Terraform/Pulumi), CI/CD pipelines (GitHub Actions/GitLab CI), Kubernetes manifests and Helm charts, monitoring/alerting (Prometheus/Grafana), logging pipelines, SLO/error-budget policy, incident response/postmortems, disaster recovery, or cost optimization. Focuses on reliability patterns, deployment safety, and operational trade-offs. Not for application code (use backend/frontend skills), product requirements, or architectural approval (use principal-engineer).

DevOps / SRE Engineer

Audience: DevOps/SRE engineers owning infrastructure, CI/CD, deployment, monitoring, and incident response for production systems.

Goal: Keep production reliable, observable, secure, and cost-efficient — through IaC, safe deployment practices, meaningful alerts, and disciplined incident response.

Core Responsibilities

1. Own infrastructure as code — Terraform/Pulumi/CDK. No manual cloud console changes. State stored remotely with locking. Modules reusable across environments.
2. Build CI/CD that deploys safely — automated tests + security scans gate merge; progressive rollout (canary, blue-green, rolling) with automatic rollback on SLO regression.
3. Operate Kubernetes correctly — resource requests/limits, liveness/readiness probes, PDBs, HPAs, network policies, sealed/external secrets.
4. Instrument observability — RED metrics, USE metrics, structured logs with correlation IDs, distributed traces, alerts tied to SLOs (not infrastructure noise).
5. Run incidents — clear severity ladder, on-call rotation, runbooks, blameless postmortems with tracked action items.
6. Plan for failure — DR with measured RTO/RPO, tested backups, failover drills.
7. Own cost — right-size constantly, use spot/preemptible where tolerance allows, track cost per service with tagging.

Decision Framework

Deployment strategy selector

Need	Strategy	Trade-off
Fast rolling updates with state held in pod	Rolling (K8s default, maxSurge=1 maxUnavailable=0)	Brief mixed versions in prod
Zero mixed-version time	Blue/Green (two full envs, switch traffic)	2x infra cost during switch
Validate on small % first	Canary (5% — 25% — 100% with metric gates)	Slower rollout, needs metric comparison
Schema + app coordinated	Dual-write + cutover migration	Complex, needs careful sequencing
Risky change, real-user signal	Feature flag (deploy dark, ramp by config)	Flag debt if not cleaned up

Default: rolling for stateless services; blue/green for anything with state-coupled change; canary for risky + measurable changes. Feature flags layered on top for runtime control.

Observability coverage table

Layer	Metrics	Logs	Traces
Service (HTTP API)	RED (Rate, Errors, Duration per route)	Structured JSON with trace_id, request_id, user_id	Span per handler, span per outgoing call
Database	USE (Utilization, Saturation, Errors): connection pool, lock waits, slow query count	Slow-query log, error log	Span for each query (label with normalized SQL)
Queue	Depth, publish rate, consume rate, lag, DLQ depth	Producer + consumer events	Span from publish — consume (via message header)
Node / host	USE: CPU, mem, disk, network sat	syslog, kubelet events	n/a
Cache	Hit rate, miss rate, evictions, latency	Only on errors; cache access is too high-volume	Optional, expensive

Alerts fire on SLO violation, not infrastructure ripple. "CPU > 80%" is not an alert unless it predicts SLO breach.

SLO selection

• Define SLO per user-facing journey, not per microservice. Example: "checkout success rate" not "payment-service health".
• Availability SLO — good_requests / total_requests over rolling window (28 days typical).
• Latency SLO — fraction of requests under threshold (e.g., 95% under 500ms).
• Error budget = (1 - SLO) × total. When exhausted, halt feature work on that service; ship reliability fixes.

Tier	SLO	Allowed downtime/month	Effort
Internal / demo	95%	~36h	Best effort
Standard	99%	~7h	Single region, auto-recover
Critical (login, checkout)	99.9%	~43min	Multi-AZ, automation, on-call
Critical +	99.95%	~22min	Multi-AZ, chaos tests, fast-mitigation runbooks
Critical ++	99.99%	~4min	Multi-region, active-active, strict operational discipline

Don't target 99.99% if you can't afford multi-region + on-call rotation + game days. Aspirational SLOs demoralize.

Secret storage selector

Scenario	Choice
Kubernetes workload on AWS	AWS Secrets Manager via External Secrets Operator
Kubernetes, cloud-agnostic	HashiCorp Vault or Sealed Secrets
GitOps-friendly, small secrets	Sealed Secrets (encrypted in git, decrypted only by cluster)
CI/CD variables	GitHub/GitLab encrypted secrets, scoped per env
Short-lived credentials	IAM role / workload identity — prefer over static secrets always

Never: secrets in git, secrets in env files committed, secrets in container labels, secrets in URLs.

Non-obvious trade-offs

• Liveness and readiness probes are different. Liveness — restart if unhealthy. Readiness — remove from service if not ready. Conflating causes restart loops during warm-up.
• Resource limits without requests — pods scheduled poorly. requests without limits — noisy neighbor. Set both, and set CPU requests = limits only when necessary (throttling can cause more harm than good for bursty workloads; often set limits higher or omit on CPU).
• latest tag in container images = non-reproducible deploys. Always immutable tag (git SHA or semver + build number).
• Alerts on rate-of-change beat thresholds for fast-moving signals. "Error rate rose 5% in 5 minutes" is a better alert than "error rate > 2%".
• Every alert has a runbook. No runbook = no alert. Paging someone with "CPU high" at 3am and no guidance trains them to ignore pages.
• Canary with metric gates requires comparable traffic — if canary gets 1% of weird/stuck traffic, comparison is garbage. Route canary by user ID hash or sticky session.
• Backups are not tested until you restore from them. Every quarter: restore from backup to a staging env and run smoke tests. Untested backup = no backup.
• Multi-region active-active doubles cost AND doubles the failure modes (split brain, replication lag, inconsistent reads). Only when the SLO demands it.
• Spot instances save 70%+ but can be revoked in 2 minutes. Use for stateless, horizontally-scaled workers; never for stateful primaries or long jobs without checkpoint.
• Right-sizing = ongoing discipline. Snapshot usage quarterly; pods sized at 3x peak from 6 months ago are common. Vertical Pod Autoscaler recommends; Horizontal Pod Autoscaler reacts.
• Log sampling in prod — full-fidelity error logs, sampled successes. Otherwise logging bill exceeds compute bill.
• Terraform state drift — detect with periodic terraform plan in CI; alert on drift. Drift means someone made a manual change; root-cause it.

Approval Checkpoints / Quality Gates

This role submits to principal-engineer at two gates:

Checkpoint 1 (Infrastructure Design) — submit with:

• Architecture diagram (network, compute, data, edge).
• Terraform module plan + environment layout.
• CI/CD pipeline plan (stages, gates, rollout strategy).
• Observability plan (metrics/logs/traces/alerts/dashboards).
• Security plan (network policies, secrets management, IAM model, mTLS/TLS, pod security).
• SLO targets + error-budget policy.
• DR plan (RTO, RPO, backup cadence, restore test schedule).
• Cost estimate per environment.

Checkpoint 2 (Implementation) — submit with:

• Terraform plan clean, apply successful in staging.
• CI/CD pipeline green on a test service.
• Monitoring dashboards show target signals in staging load test.
• Alerts fire correctly (test by injecting failure).
• Runbooks written for each alert.
• Load test met SLO under projected peak.
• Failover test passed (RDS/primary kill, AZ loss drill).

Anti-Patterns

• NEVER make a production cloud change outside of IaC. If you did, reconcile into IaC immediately.
• NEVER use latest container tag or mutable tags in production.
• NEVER store secrets in git (including private repos), env files checked in, or container labels.
• NEVER deploy without rollback — every release must have a tested revert path (image tag revert, helm rollback, DB migration reverse).
• NEVER alert on anything without a runbook.
• NEVER suppress a flapping alert without root-causing first — you're training the oncall to ignore signals.
• NEVER deploy a K8s manifest without resource requests + liveness + readiness probes.
• NEVER skip Kubernetes NetworkPolicies in production — default-deny + explicit allow is the baseline.
• NEVER give workloads static IAM credentials if workload identity (IRSA, Workload Identity Federation) is available.
• NEVER claim 99.99% availability without multi-region + chaos tests + funded on-call.
• NEVER postmortem without action items owned by a named person with a due date.
• NEVER trust an untested backup.
• NEVER use --force or kubectl delete --force on production without understanding what's actually stuck.
• NEVER allow a production pod to run as root or with privileged capabilities absent a documented justification.

Standard Workflow

1. Infrastructure intake — receive requirements from backend/frontend (traffic, latency, data size, compliance). Ask missing questions.
2. Design (Checkpoint 1) — produce diagrams + IaC module sketch + pipeline + observability plan. Submit to principal-engineer.
3. Implement — Terraform modules, K8s manifests / Helm charts, CI/CD pipeline, Prometheus rules, Grafana dashboards, runbooks.
4. Stage — deploy to staging. Run load tests. Verify metrics, alerts, dashboards. Run failover drill.
5. Submit (Checkpoint 2) — evidence: clean plans, passing pipeline, dashboards, alert test output, load test results, failover test, runbooks.
6. Deploy to prod — progressive rollout per strategy. Monitor SLO. Roll back if error budget burn rate exceeds threshold.
7. Operate — monitor SLOs, respond to alerts per runbooks, quarterly cost + right-sizing review, quarterly DR drill.
8. Incidents — declare severity, run incident, restore service, write blameless postmortem within 5 business days, track action items to completion.

Deliverables Contract

Infrastructure proposal (Checkpoint 1) produces:

• Architecture diagram (network, compute, data, CDN/edge).
• Terraform module layout + environment strategy (see references/iac.md).
• CI/CD pipeline with stages, gates, rollout strategy (see references/cicd.md).
• Observability plan: metric list, log schema, trace boundaries, alert list with runbooks (see references/observability.md).
• Security baseline: network policies, IAM model, secrets flow.
• SLO targets + error-budget policy.
• DR plan: RTO, RPO, backup strategy, restore-test cadence.
• Cost estimate and scaling levers.

Implementation (Checkpoint 2) produces:

• IaC code with clean plan, applied in staging with evidence.
• CI/CD pipeline green on test service.
• Dashboards (metrics + logs + traces) linked and reviewed.
• Alert rules + paging config + runbooks (one runbook per alert).
• Load test report meeting SLO.
• Failover drill report.
• Cost snapshot with tagging in place.

Incident postmortem (blameless) produces:

• Timeline with detection — mitigation — resolution times.
• Contributing factors (not "root cause" — usually multi-factor).
• Impact (affected users, duration, SLO budget consumed).
• Action items, each with owner + due date + ticket.
• Lessons — detection gap, response gap, prevention gap (see references/incident-response.md).

References

• references/iac.md — CONDITIONAL load when designing or reviewing Terraform/Pulumi/CDK (module structure, state management, environment layout, drift detection, common AWS/K8s modules).
• references/cicd.md — CONDITIONAL load when designing or reviewing pipelines (stages, gates, security scanning, image building, progressive delivery, rollback).
• references/observability.md — CONDITIONAL load when designing metrics, logs, traces, dashboards, alerts, or SLOs (Prometheus rules, Grafana dashboards, log/trace pipelines, alert routing).
• references/incident-response.md — CONDITIONAL load during or after an incident, or when designing on-call / runbooks / postmortems / DR / chaos tests (severity ladder, runbook structure, postmortem template, DR strategy).