loom-error-handling

# Error Handling ## Overview Cross-language error-handling reference. The recurring mistakes are not syntactic: swallowing errors, retrying non-idempotent operations, losing the cause chain when wrapping, leaking internals in API responses, and log-and-rethrow double logging. This skill leads with the decision rules an expert applies, then shows the minimum code to implement each. ## Agent Delegation - **loom-senior-software-engineer** (Opus) — DEFAULT. Error architecture, strategy choice, secure handling (no info leak), infra resilience (retry/circuit breaker/fallback). - **loom-software-engineer** (Sonnet) — ONLY boilerplate error types / unit tests following an established pattern. ## Error Taxonomy — Classify First Every failure is exactly one of three kinds; the kind dictates handling: | Kind | Meaning | Handle by | | --- | --- | --- | | **Expected / recoverable** | Invalid input, not-found, transient network/rate-limit | Return a typed error (`Result`/checked exception); retry only if transient AND idempotent | | **Bug / programmer error** | Broken invariant, unreachable state, index OOB | Fail loud: `panic!`/assert/throw. Do NOT try to recover — it hides the bug | | **Fatal / environmental** | OOM, disk full, config missing at startup | Fail fast at the boundary; crash cleanly, let the supervisor restart | ⚠ Do not model expected failures as panics/exceptions-for-control-flow, and do not swallow bugs into a recoverable path. Rust encodes this split directly: `Result` = recoverable, `panic!` = bug. In exception languages, keep the split by convention: validation → typed exception you catch; invariant break → let it propagate/crash. ## Rust: Result/Option & the anyhow-vs-thiserror Rule **Decision rule:** - **`thiserror`** → **libraries** and any code whose caller must *match on* and react to specific error variants. Gives typed, exhaustive enums with `#[from]` conversions. - **`anyhow`** (or `eyre`) → **application/binary** top layers where you only need to *propagate, add context, and report*. One `anyhow::Error` type, cheap `.context()`, backtraces. - Mixed is normal: libs export `thiserror` enums; `main`/handlers use `anyhow` and add context as errors bubble up. ```rust use thiserror::Error; #[derive(Error, Debug)] pub enum AppError { #[error("validation failed: {0}")] Validation(String), #[error("resource not found: {resource_type} id={id}")] NotFound { resource_type: String, id: String }, #[error("database error")] Database(#[from] sqlx::Error), // free `?` conversion, preserves source } // anyhow at the app layer: `?` + context to build a cause chain. use anyhow::{Context, Result}; fn process_order(id: &str) -> Result<Order> { let order = fetch_order(id).with_context(|| format!("fetching order {id}"))?; validate(&order).context("order validation")?; Ok(order) } ``` Combinators over match ladders: - `?` propagates and auto-converts via `From`/`#[from]`. `map_err` adapts an error type; `.context()`/`.with_context()` (lazy, use for allocating messages) wrap while preserving `source()`. - `Option`: `ok_or_else` → `Result`; `?` works on `Option` in `Option`-returning fns. Prefer `.ok_or_else(|| ...)` (lazy) over `.ok_or(expensive())`. - ⚠ Never `.unwrap()`/`.expect()` on recoverable errors in library/production paths — that converts a handleable error into a crash. `expect("invariant: X")` is acceptable only for genuine bugs where the message documents the invariant. ## Cause Chains — Wrap, Don't Erase Wrapping adds context; it must **preserve the original cause** so a reader can trace root cause. Every language has the idiom — use it: | Language | Wrap-with-cause | Inspect chain | | --- | --- | --- | | Rust | `.with_context(...)` / `#[source]` / `#[from]` | `err.source()` / `{:?}` | | Go | `fmt.Errorf("...: %w", err)` | `errors.Is` / `errors.As` / `Unwrap` | | Python | `raise New(...) from err` | `__cause__` / traceback | | JS/TS | `new Error(msg, { cause: err })` | `err.cause` | ```go // Go: %w preserves the chain so errors.Is/As work up the stack. if err != nil { return fmt.Errorf("process order %s: %w", id, err) } // caller: if errors.Is(err, sql.ErrNoRows) { ... } ``` ```python # Python: `from err` keeps __cause__; bare `raise New(...)` LOSES the original. try: order = fetch_order(id) except DatabaseError as e: raise ServiceError("failed to process order", code="ORDER_FAILED", details={"order_id": id}) from e # ← preserve cause ``` ⚠ `%v` (Go) or f-string interpolation of an error into a *new* message string does NOT preserve the typed chain — `errors.Is`/`isinstance` checks then break. Use `%w` / `from` / `cause`. ## Anti-Patterns (the ones that actually bite) - **Swallowing** — empty `catch {}`, bare `except:`, `if err != nil { }`, `let _ = op();`. At minimum log; usually handle or propagate. Bare `except:` also eats `KeyboardInterrupt`/`CancelledError` — always `except Exception` or narrower. - **Log-and-rethrow** — logging an error then re-throwing it logs the same failure at every frame → noisy duplicate stacks. **Handle OR log-and-swallow OR propagate — pick one.** Rule: log where you *handle* (usually one boundary), propagate everywhere else. - **Exceptions for control flow** — using try/except to implement normal branching is slow and hides logic; check the condition instead. (Python's EAFP is fine for genuinely exceptional cases, not for expected branches.) - **Catch-all too early** — a broad catch near the leaf hides the specific error and its recovery opportunity. Catch narrow and near, or broad and at the boundary — not broad and deep. - **Retrying non-idempotent ops** — see below. - **Leaking internals** — stack traces / SQL / paths in API responses; see API section. ## Recovery Strategies Choose by failure kind: | Strategy | Use when | Caution | | --- | --- | --- | | **Retry + backoff + jitter** | Transient AND **idempotent** | Never retry a non-idempotent write without an idempotency key | | **Fallback** | A degraded alternative exists (cache, default, secondary) | Fallback data must be safe to serve; log the degradation | | **Circuit breaker** | Repeated failures to a dependency; prevent cascade | Tune threshold/timeout; expose state for observability | | **Fail fast** | Bad config, missing dep at startup, unrecoverable | Crash cleanly at the boundary, don't limp on | ⚠ **Retry only idempotent operations.** GET/PUT/DELETE are typically idempotent; a raw POST "create" is not — a retried create makes duplicates. Make writes idempotent with a client-supplied idempotency key, or don't retry them. Also: retry only *transient* errors (timeout, 429, 503) — retrying a 400/permission error just wastes time and amplifies load. **Backoff must have jitter** to avoid the thundering-herd where all clients retry in lockstep and re-synchronize the overload. `delay = min(base * 2^attempt, cap)`; then randomize (full jitter: `random(0, delay)`). ```rust // Cap + exponential + jitter; retry only on retryable errors. let mut attempt = 0u32; loop { match op().await { Ok(v) => break Ok(v), Err(e) if !e.is_retryable() || attempt >= max => break Err(e), Err(_) => { let backoff = (base_ms * 2u64.pow(attempt)).min(cap_ms); let delay = rand::thread_rng().gen_range(0..=backoff); // full jitter tokio::time::sleep(Duration::from_millis(delay)).await; attempt += 1; } } } ``` ```python # Circuit breaker: CLOSED → (failures ≥ threshold) → OPEN → (after timeout) → HALF_OPEN → CLOSED/OPEN class CircuitBreaker: def __init__(self, threshold=5, recovery_timeout=30.0): self.threshold, self.recovery_timeout = threshold, recovery_timeout self.state, self.failures, self.opened_at = "closed", 0, 0.0 def call(self, op): if self.state == "open": if time.time() - self.opened_at > self.recovery_timeout: self.state = "half_open" # probe with a single call else: raise CircuitOpenError() try: r = op(); self.failures = 0; self.state = "closed"; return r except Exception: self.failures += 1 if self.failures >= self.threshold: self.state, self.opened_at = "open", time.time() raise ``` ## API Error Responses (RFC 9457 / 7807 problem+json) Design rules for HTTP APIs: - **Stable machine-readable code** the client can branch on (`"code": "ORDER_NOT_FOUND"`), independent of the human `detail` message. Never make clients string-match on prose. - **Never leak internals** — no stack traces, SQL, file paths, or exception class names in 5xx responses. Show `detail` for 4xx (client can act on it), hide it for 5xx. - Content type `application/problem+json`; fields: `type`, `title`, `status`, `detail`, `instance` (+ your `code` and any safe extensions). - Map error *kind* → status: validation→400, auth→401, forbidden→403, not-found→404, conflict→409, rate-limit→429, dependency down→503, bug→500. ```python class ProblemDetails(BaseModel): type: str = "about:blank" title: str status: int code: str | None = None detail: str | None = None instance: str | None = None @app.exception_handler(AppError) async def handler(request, error: AppError): status = {ValidationError: 400, NotFoundError: 404, ServiceError: 503}.get(type(error), 500) problem = ProblemDetails( title=type(error).__name__, status=status, code=error.code, detail=str(error) if status < 500 else None, # hide internals on 5xx instance=str(request.url.path), ) return JSONResponse(status, problem.model_dump(exclude_none=True), headers={"Content-Type": "application/problem+json"}) ``` ```rust // Rust/axum: map variants to status; collapse internal errors to a generic 500. impl IntoResponse for AppError { fn into_response(self) -> Response { let (status, detail) = match self { AppError::Validation(m) => (StatusCode::BAD_REQUEST, Some(m)), AppError::NotFound { .. } => (StatusCode::NOT_FOUND, Some(self.to_string())), AppError::Database(_) => (StatusCode::INTERNAL_SERVER_ERROR, None), // never expose }; (status, Json(json!({ "title": status.canonical_reason(), "status": status.as_u16(), "detail": detail }))).into_response() } } ``` ## Security-Aware Handling - **Production vs dev detail** — full error type + backtrace only when `ENVIRONMENT != "production"`; production returns a generic message. Gate on env, not on a debug flag a client can flip. - **Sanitize before logging** — redact emails, SSNs, card numbers, tokens/passwords from messages *before* they hit logs. An error string built from user input can carry PII or secrets into your log store. - **Rate-limit error logs** — a hot failure path can flood logs (cost + DoS); throttle per error-key (log first occurrence + a periodic count). - **Uniform auth errors** — return the same 401/403 for "user not found" and "wrong password" to avoid user enumeration. ```python SENSITIVE = [r'[\w.%+-]+@[\w.-]+\.\w{2,}', r'\b\d{3}-\d{2}-\d{4}\b', r'\b(?:\d{4}[-\s]?){3}\d{4}\b', r'password["\']?\s*[:=]\s*\S+'] def sanitize(msg: str) -> str: for p in SENSITIVE: msg = re.sub(p, '[REDACTED]', msg, flags=re.IGNORECASE) return msg ``` ## Data Pipeline / Batch Errors Batch jobs need **partial-failure** handling: don't let one bad record kill 10k good ones. Collect failures with their index + payload, route to a dead-letter queue, and surface an error summary — but fail the whole job if the failure *rate* crosses a threshold (a 40% failure rate is a systemic problem, not bad records). ```python @dataclass class BatchResult(Generic[T]): ok: list[T]; failed: list[dict]; summary: dict[str, int] def process_batch(items, fn, *, continue_on_error=True) -> BatchResult: ok, failed, counts = [], [], {} for i, item in enumerate(items): try: ok.append(fn(item)) except Exception as e: counts[type(e).__name__] = counts.get(type(e).__name__, 0) + 1 failed.append({"index": i, "item": item, "error": str(e)}) logging.error("item %d failed: %s", i, e) if not continue_on_error: raise return BatchResult(ok, failed, counts) # route failed → dead_letter_queue; alert if len(failed)/len(items) > threshold ``` ## Verification Checklist - [ ] Every failure classified: recoverable (typed error) vs bug (panic/assert) vs fatal (fail-fast) — no expected failures modeled as panics, no bugs swallowed into recoverable paths