google-sre

name	google-sre
description	Apply Google's Site Reliability Engineering methodology. Emphasizes error budgets, SLO-driven operations, toil elimination, and blameless postmortems. Use when building and operating reliable services at scale.
tags	sre, reliability, monitoring, slo, error-budget, incident-response, toil, automation, observability, on-call

Google Site Reliability Engineering (SRE)⁠‍⁠​‌​‌​​‌‌‍​‌​​‌​‌‌‍​​‌‌​​​‌‍​‌​​‌‌​​‍​​​​​​​‌‍‌​​‌‌​‌​‍‌​​​​​​​‍‌‌​​‌‌‌‌‍‌‌​​​‌​​‍‌‌‌‌‌‌​‌‍‌‌​‌​​​​‍​‌​‌‌‌‌‌‍​‌​​‌​‌‌‍​‌‌​‌​​‌‍‌​‌​‌‌‌​‍​​‌​‌​​​‍‌‌‌​‌​‌‌‍​​​​​‌​​‍‌​‌‌‌‌‌‌‍‌​​​​​‌​‍​‌​‌​‌‌​‍​​​​‌​‌​‍​​​​​‌​‌⁠‍⁠

Overview

Site Reliability Engineering (SRE) is Google's approach to running production systems. It applies software engineering principles to operations, treating reliability as a feature that can be measured, budgeted, and engineered.

References

• Book: "Site Reliability Engineering: How Google Runs Production Systems" (O'Reilly, 2016)
• Workbook: "The Site Reliability Workbook" (O'Reilly, 2018)
• Online: https://sre.google/

Core Philosophy

"Hope is not a strategy."

"SRE is what happens when you ask a software engineer to design an operations function."

"Reliability is the most important feature."

SRE balances the tension between development velocity and system reliability using measurable objectives and error budgets.

Key Concepts

The Service Level Hierarchy

SLI (Service Level Indicator)
    ↓ Quantitative measure of service
    ↓ Example: "Request latency < 100ms"

SLO (Service Level Objective)  
    ↓ Target value for SLI
    ↓ Example: "99.9% of requests < 100ms"

SLA (Service Level Agreement)
    ↓ Contract with consequences
    ↓ Example: "If SLO missed, credits issued"

Error Budget = 100% - SLO
    Example: 99.9% SLO = 0.1% error budget = 43 minutes/month downtime

Error Budget Philosophy

Error Budget Remaining?
        │
    ┌───┴───┐
    │       │
   YES      NO
    │       │
    ↓       ↓
 Ship new   Focus on
 features   reliability

Design Principles

1. Embrace Risk: 100% reliability is wrong target; it's too expensive.

2. Error Budgets: Explicit budget for unreliability enables velocity.

3. Eliminate Toil: Automate repetitive operational work.

4. Simplicity: Simple systems are more reliable.

When Implementing

Always

• Define SLIs before launching a service
• Set SLOs based on user needs, not engineering pride
• Track error budget consumption
• Measure and reduce toil
• Conduct blameless postmortems
• Automate incident response where possible

Never

• Set SLOs at 100% (it's impossible and wrong)
• Ignore SLO violations
• Blame individuals for outages
• Accept toil as "just how things are"
• Skip postmortems for "small" incidents

Prefer

• Automation over manual processes
• Gradual rollouts over big-bang deploys
• Monitoring over hoping
• Documentation over tribal knowledge
• Proactive work over reactive firefighting

Implementation Patterns

Defining SLIs and SLOs

# slo_definitions.py
# Define service level indicators and objectives

from dataclasses import dataclass
from enum import Enum
from typing import Optional

class SLIType(Enum):
    AVAILABILITY = "availability"
    LATENCY = "latency"
    THROUGHPUT = "throughput"
    ERROR_RATE = "error_rate"
    FRESHNESS = "freshness"

@dataclass
class SLI:
    """Service Level Indicator - what we measure"""
    name: str
    type: SLIType
    description: str
    good_event_query: str      # Events that are "good"
    total_event_query: str     # All events
    
    def calculate(self, good_count: int, total_count: int) -> float:
        if total_count == 0:
            return 1.0
        return good_count / total_count

@dataclass
class SLO:
    """Service Level Objective - our target"""
    sli: SLI
    target: float              # e.g., 0.999 for 99.9%
    window_days: int           # Rolling window
    

     () -> :
        
          - .target
    
     () -> :
        
        errors_used =  - current_sli
         .error_budget == :
             
         (,  - (errors_used / .error_budget))



availability_sli = SLI(
    name=,
    =SLIType.AVAILABILITY,
    description=,
    good_event_query=,
    total_event_query=
)

availability_slo = SLO(
    sli=availability_sli,
    target=,        
    window_days=       
)

Error Budget Tracking

# error_budget.py
# Track and alert on error budget consumption

import time
from dataclasses import dataclass
from typing import List
from datetime import datetime, timedelta

@dataclass
class ErrorBudgetTracker:
    slo: 'SLO'
    window_seconds: int
    
    def __init__(self, slo: 'SLO'):
        self.slo = slo
        self.window_seconds = slo.window_days * 24 * 60 * 60
        self.events: List[tuple] = []  # (timestamp, is_good)
    
    def record_event(self, is_good: bool):
        """Record an event"""
        now = time.time()
        self.events.append((now, is_good))
        self._prune_old_events(now)
    
    def _prune_old_events(self, now: float):
        """Remove events outside window"""
        cutoff = now - self.window_seconds
        self.events = [(t, g) for t, g in self.events if t > cutoff]
    
    def () -> :
        
          .events:
             
        good = (  _, is_good  .events  is_good)
         good / (.events)
    
     () -> :
        
         .slo.budget_remaining(.current_sli()) * 
    
     () -> [timedelta]:
        
        remaining = .budget_remaining_percent()
         remaining <= :
             timedelta()
        
        
        
           
    
     () -> :
        
         .budget_remaining_percent() <   



 ():
    
    remaining = tracker.budget_remaining_percent()
    
     remaining < :
         
     remaining < :
         
     remaining < :
         
     remaining < :
         
    :
         

Toil Measurement and Elimination

# toil_tracker.py
# Measure and track operational toil

from dataclasses import dataclass
from enum import Enum
from typing import List, Dict
from datetime import datetime, timedelta

class ToilCategory(Enum):
    MANUAL = "manual"           # Could be automated
    REPETITIVE = "repetitive"   # Done frequently
    TACTICAL = "tactical"       # Reactive, not strategic
    NO_VALUE = "no_value"       # Doesn't improve service
    SCALES_LINEARLY = "scales"  # Grows with service size

@dataclass
class ToilTask:
    name: str
    categories: List[ToilCategory]
    time_spent_minutes: int
    frequency_per_week: float
    automation_possible: bool
    automation_effort_days: float
    
    @property
    def weekly_toil_hours(self) -> float:
        return (self.time_spent_minutes * self.frequency_per_week) / 60
    
    @property
    def automation_roi_weeks(self) -> :
        
          .automation_possible:
             ()
        
        effort_hours = .automation_effort_days * 
         effort_hours / .weekly_toil_hours


 :
    
    team_size: 
    hours_per_week:  = 
    max_toil_percent:  = 
    

     () -> :
         .team_size * .hours_per_week * .max_toil_percent
    
     () -> :
         current_toil_hours > .max_toil_hours_per_week


 :
     ():
        .budget = budget
        .tasks: [, ToilTask] = {}
    
     ():
        .tasks[task.name] = task
    
     () -> :
         (t.weekly_toil_hours  t  .tasks.values())
    
     () -> :
        max_hours = .budget.team_size * .budget.hours_per_week
         (.total_weekly_toil() / max_hours) * 
    
     () -> [ToilTask]:
        
        automatable = [t  t  .tasks.values()  t.automation_possible]
         (automatable, key= t: t.automation_roi_weeks)
    
     () -> :
        report = []
        report.append()
        report.append()
        report.append()
        report.append()
        report.append()
         task  .automation_priorities()[:]:
            report.append()
         .join(report)

Blameless Postmortem Template

# Postmortem: [Incident Title]

**Date**: YYYY-MM-DD
**Authors**: [Names]
**Status**: Draft | In Review | Complete
**Severity**: P0 | P1 | P2 | P3

## Summary

[2-3 sentences describing what happened, impact, and resolution]

## Impact

- **Duration**: X hours Y minutes
- **Users affected**: N users / X% of traffic
- **Revenue impact**: $X (if applicable)
- **Error budget consumed**: X%

## Timeline (all times UTC)

| Time | Event |
|------|-------|
| HH:MM | First alert fired |
| HH:MM | On-call engaged |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Service fully recovered |

## Root Cause

[Technical explanation of what caused the incident]

## Resolution

[What was done to resolve the incident]

## Detection

- How was the incident detected?
- Could we have detected it sooner?
- What monitoring would have helped?

## Lessons Learned

### What went well

- [Things that worked]

### What went wrong

- [Things that didn't work]

### Where we got lucky

- [Things that could have made it worse]

## Action Items

| Action | Type | Owner | Due Date | Status |
|--------|------|-------|----------|--------|
| Add monitoring for X | Detect | @name | YYYY-MM-DD | TODO |
| Implement circuit breaker | Mitigate | @name | YYYY-MM-DD | TODO |
| Update runbook | Process | @name | YYYY-MM-DD | TODO |

## Supporting Information

- Relevant logs, graphs, or documentation
- Links to related incidents

On-Call Rotation Best Practices

# oncall.py
# On-call rotation management

from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import List, Optional

@dataclass
class OnCallShift:
    engineer: str
    start: datetime
    end: datetime
    
    @property
    def duration_hours(self) -> float:
        return (self.end - self.start).total_seconds() / 3600

@dataclass
class OnCallPolicy:
    """Google SRE on-call best practices"""
    
    # Shift structure
    max_shift_hours: int = 12          # No more than 12 hours
    min_time_between_shifts: int = 12  # At least 12 hours rest
    max_incidents_per_shift: int = 2   # Escalate if exceeded
    
    # Team structure
    min_team_size: int = 8             # For sustainable rotation
    secondary_oncall: bool = True      # Always have backup
    
    # Compensation
    time_off_per_incident: float = 0.5 # Hours of comp time
    
     () -> []:
        
        violations = []
        
         shift.duration_hours > .max_shift_hours:
            violations.append(
                
            )
        
        
         prev  previous_shifts:
             prev.engineer == shift.engineer:
                gap = (shift.start - prev.end).total_seconds() / 
                 gap < .min_time_between_shifts:
                    violations.append(
                        
                    )
        
         violations

     () -> :
        
         incidents_handled * .time_off_per_incident

Mental Model

Google SRE asks:

1. What's the SLO? Reliability target based on user needs
2. What's the error budget? How much unreliability can we afford?
3. Is this toil? Manual, repetitive, automatable, no lasting value?
4. What does the postmortem say? Learn from failures, don't blame
5. Can we ship this safely? Gradual rollout with monitoring

Signature SRE Moves

• Error budgets to balance reliability and velocity
• SLI/SLO/SLA hierarchy for clear targets
• Toil tracking and elimination
• Blameless postmortems
• On-call that doesn't burn out engineers
• Automation as first response to toil