classBulkhead {
private active = 0;
privatequeue: Array<() =>void> = [];
constructor(privatemaxConcurrent: number = 10,
privatemaxQueue: number = 100) {}
async execute<T>(fn: () =>Promise<T>): Promise<T> {
if (this.active >= this.maxConcurrent) {
if (this.queue.length >= this.maxQueue) {
thrownewBulkheadFullError();
}
awaitnewPromise<void>(resolve =>this.queue.push(resolve));
}
this.active++;
try {
returnawaitfn();
} finally {
this.active--;
const next = this.queue.shift();
if (next) next();
}
}
}
Retry Strategy Matrix
Error Type
Strategy
Base Delay
Max Retries
Jitter
Network timeout
Exponential backoff
100ms
3
Yes
Rate limited (429)
Respect Retry-After
Header value
5
No
Service unavailable (503)
Exponential backoff
1s
5
Yes
Connection refused
Linear backoff
5s
10
Yes
Authentication (401)
No retry
—
0
—
Bad request (400)
No retry
—
0
—
Conflict (409)
Retry with fresh data
500ms
3
No
Error Classification
Type
Detection
Response
Transient
Timeout, 503, network error
Retry with backoff
Permanent
400, 401, 403, 404
Fail fast, no retry
Partial
Some items succeed, some fail
Degrade gracefully
Cascading
Multiple services failing
Circuit break, shed load
Graceful Degradation Strategies
Service dependency fails?
├── Is there a cached response?
│ └── YES → Return cached (with stale indicator)
├── Is there a default/fallback value?
│ └── YES → Return default
├── Is this feature optional?
│ └── YES → Hide feature, continue
└── Is this critical path?
└── YES → Return error with clear message
Strategy
Use When
Example
Cached fallback
Data changes slowly
User profile from cache
Default value
Reasonable default exists
Default settings
Feature toggle
Feature is non-critical
Hide recommendations
Read-only mode
Write path fails
Allow browsing, disable checkout
Static response
Service is down
Maintenance page
SLO Framework
Define SLIs (Service Level Indicators)
SLI
Measurement
Good Threshold
Availability
Successful requests / total requests
> 99.9%
Latency
Response time at percentile
p95 < 200ms
Throughput
Requests per second
> baseline
Error rate
Failed requests / total requests
< 0.1%
Error Budget Calculation
SLO = 99.9% availability
Error Budget = 100% - 99.9% = 0.1%
Per month: 30 days × 24h × 60min = 43,200 minutes
Allowed downtime: 43,200 × 0.001 = 43.2 minutes/month
If budget remaining > 50% → Ship features aggressively
If budget remaining < 25% → Focus on reliability
If budget exhausted → Feature freeze, fix reliability
