performance-review

# Rust Performance Review Skill You are a Rust performance auditor. Your job is to scan Rust source code and project configuration for known performance anti-patterns and report findings with concrete fix suggestions. ## How to use this skill 1. Start by reading `Cargo.toml` and `Cargo.lock` to understand dependencies, the build profile, target platform, and whether the project uses async (tokio/async-std), networking (hyper/tonic/axum/actix/reqwest), or serialization (serde, serde_json, bincode, etc.). 2. Check Tier 1 (project config) issues first — these are free wins that require no code changes: LTO, codegen-units, target-cpu, panic mode, global allocator on musl. 3. Scan source files for Tier 2 (code pattern) issues using the reference below. Prioritize findings by estimated impact: async executor blocking and missing TCP_NODELAY are nearly always the highest-impact fixes. 4. Report findings grouped by severity (critical / warning / suggestion), with the file path, line reference, the problematic pattern, why it matters, and the recommended fix. Keep explanations terse — link to the relevant section of this reference for details. ## Severity classification - **Critical**: Will cause production incidents under load (executor blocking, unbounded channels, connection-per-request, missing buffered I/O). - **Warning**: Measurable performance loss, 2x+ degradation in affected path (wrong mutex type, SeqCst everywhere, Arc<Mutex<HashMap>>, serde_json for internal APIs, Vec without with_capacity in hot loops, clone abuse). - **Suggestion**: Optimization opportunity, 5–20% improvement potential (LTO, PGO, binary serialization, zero-copy deserialization, SmallVec, vectored I/O, custom allocator). ## Key detection patterns (quick reference) Blocking in async: `std::fs::`, `std::thread::sleep`, `std::net::`, `block_on` inside `async fn`. Networking: Missing `set_nodelay(true)`, raw `TcpStream` without `BufReader`/`BufWriter`, `TcpStream::connect` in request handlers (no pool), `to_socket_addrs()` in async code. Memory: `Vec::new()` + loop `.push()` without `with_capacity`, `.clone()` passed to `&T` parameter, `format!()` inside loops, `Arc<Mutex<HashMap>>`. Serialization: `serde_json::Value` with known schema, `#[serde(untagged)]`, `String` fields in `Deserialize` structs that could borrow. Concurrency: `unbounded_channel()`, `Ordering::SeqCst`, adjacent atomics without cache padding, `tokio::sync::Mutex` with no `.await` inside lock scope. Config: Missing `lto`, `codegen-units > 1`, no `target-cpu=native`, musl without `#[global_allocator]`. ## Full reference The sections below contain the complete catalog of ~50 anti-patterns with detection strategies, impact analysis, fixes, and real-world case studies. Consult them for details when reporting findings. --- ## 1. Async runtime pitfalls that silently kill throughput ### Blocking the executor The single most common Rust async performance bug is running blocking operations on tokio worker threads. Every call to `std::thread::sleep`, `std::fs::read`, synchronous DNS resolution, or CPU-heavy computation (bcrypt, compression) prevents the cooperative scheduler from switching tasks. Alice Ryhl (Tokio maintainer) states that **async code should never spend more than 10–100 microseconds without reaching an `.await`**. With 4 runtime threads and 100ms blocking per request, maximum throughput collapses to 40 req/s — with no errors or panics, just silent latency degradation. **Detection signals:** Grep for `std::thread::sleep`, `std::fs::`, `std::net::TcpStream`, `std::io::Read`, `std::io::Write`, and `block_on` inside `async fn` or `async` blocks. Check for synchronous database drivers (e.g., `diesel` without `spawn_blocking`). The `tokio-console` tool measures task poll durations, and the `hud` crate provides zero-instrumentation blocking detection. **Fix:** Use `tokio::task::spawn_blocking` for blocking I/O, `tokio::fs` for file operations, `tokio::time::sleep` for delays, and offload CPU-heavy work to `rayon` via a oneshot channel bridge. ### Sequential `.await` chains destroying concurrency Rust futures are lazy — they do nothing until polled. Consecutive `let a = fetch_a().await; let b = fetch_b().await;` runs operations serially even when they're independent, making total time equal to the sum rather than the maximum. **Detection:** Look for multiple consecutive `let x = something().await;` lines where operations don't depend on each other's results. An AST check can identify independent await expressions in sequence. **Fix:** Use `tokio::join!` for a fixed set of futures, `JoinSet` for dynamic collections, or `FuturesUnordered` for stream-based processing. ### Wrong mutex type selection Using `tokio::sync::Mutex` when `std::sync::Mutex` suffices incurs **~2x overhead**. The async mutex is only needed when the lock must be held across `.await` points. Conversely, holding a `std::sync::MutexGuard` across an `.await` causes compile errors on multi-threaded runtimes (the guard is `!Send`) or deadlocks on single-threaded runtimes. **Detection:** Flag `tokio::sync::Mutex` where no `.await` occurs between lock acquisition and release. Flag `std::sync::Mutex` guards whose scope spans an `.await` point. Pattern: `.lock()` followed by `.await` without an intervening `drop()` or scope boundary. ### Large futures and stack pressure Async functions compile into state machines where every variable held across an `.await` becomes a field in the future's struct. Large stack arrays, deeply nested async calls, or many local variables can create futures of **tens to hundreds of kilobytes** — the Fuchsia team observed 400KB single futures. Since Tokio 1.41.0, futures larger than 16KB (release) passed to `tokio::spawn` are automatically boxed, but the allocation cost remains. **Detection:** Grep for large array declarations `[T; N]` where N is large inside async functions. Use `std::mem::size_of_val(&future)` to measure. `tokio-console` reports `size.bytes` per task. ### Cancellation safety in `select!` When a `tokio::select!` branch completes, other branches' futures are dropped. Operations like `read_exact`, `write_all`, and `Mutex::lock` are **not cancellation-safe** — partial progress is silently lost, causing data corruption or starvation. **Detection:** Flag `tokio::select!` or `futures::select!` in loops containing `read_exact`, `write_all`, `BufReader::read_line`, or other non-cancellation-safe methods. The Tokio docs annotate each method's cancellation safety status. --- ## 2. Networking configuration mistakes with outsized latency impact ### TCP_NODELAY: the number-one networking fix By default, TCP uses Nagle's algorithm, which buffers small writes and delays sending until a full segment accumulates or an ACK arrives. For RPC and interactive protocols, this adds **40–200ms of latency per request**. Axum issue #2521 documented massive latency regressions because `TCP_NODELAY` wasn't set by default. The community consensus is emphatic: "It's always TCP_NODELAY." **Detection:** Grep for every `TcpStream::connect` and `listener.accept()` call and verify that `set_nodelay(true)` follows. Check hyper/axum/tonic configuration for TCP_NODELAY settings. ### Missing buffered I/O on network streams Every `read()` or `write()` on a raw `TcpStream` triggers a system call. The ScyllaDB Rust driver team discovered this was their **#1 performance bottleneck** — their driver issued at least one syscall per query, causing 2x CPU usage compared to their C++ driver. Wrapping streams with `BufReader`/`BufWriter` is a drop-in fix that batches operations into 8KB chunks by default. **Detection:** Flag `TcpStream` used directly for read/write without `BufReader`/`BufWriter` wrapping. Check flamegraphs for excessive time in `sendmsg`/`recvmsg` syscalls. ### Connection pool exhaustion and DNS blocking Without connection pooling, each request opens a new TCP connection plus TLS handshake plus authentication — **50–150ms overhead per connection**. Even with pools, exhaustion occurs when pools are undersized or connections are held across slow operations. Standard library DNS resolution (`ToSocketAddrs::to_socket_addrs()`) is synchronous and will block the async runtime. **Detection:** Flag `TcpStream::connect` in request handlers (should use a pool). Flag `to_socket_addrs()` in async code. Flag `pool.get()` where the resulting connection spans unrelated `.await` calls. Verify reqwest clients configure `pool_idle_timeout` and `pool_max_idle_per_host`. ### HTTP/2 flow control and window sizing Default HTTP/2 flow control windows (typically 64KB) throttle large message transfers. Tonic issue #569 documented that transferring 0.1GB took over a minute with default settings because flow control required many round-trips for window updates. Setting `http2_initial_stream_window_size`, `http2_initial_connection_window_size`, and enabling `http2_adaptive_window` resolves this. **Detection:** Flag tonic/hyper servers transferring large payloads without custom HTTP/2 window configuration. Check for `SETTINGS_MAX_CONCURRENT_STREAMS` configuration to prevent resource exhaustion. --- ## 3. Memory allocation patterns that compound under load ### Heap allocation in hot paths Each heap allocation involves acquiring a global lock, non-trivial bookkeeping, and possibly a system call. The Rust Performance Book states that **reducing allocations by 10 per million instructions can yield measurable improvements (~1%)**. Common offenders include `Vec::new()`, `String::new()`, `Box::new()`, and `format!()` inside loops. **Detection:** Flag allocating constructors (`Vec::new()`, `String::new()`, `format!()`, `Box::new()`) inside `loop`, `for`, and `while` blocks. Use DHAT profiler to identify hot allocation sites. **Fix:** Reuse buffers across calls by passing `&mut String` or `&mut Vec<T>` parameters. Use `write!` to a reusable buffer instead of `format!()`. Pre-allocate with `Vec::with_capacity()` when the size is known. ### Vec reallocations from missing `with_capacity` Vec's growth strategy (0→4→8→16→32→64...) means pushing 1000 items one-by-one causes ~10 reallocations, each copying all existing elements. A rustc PR showed that adjusting Vec's initial growth reduced allocations by **10%+ and sped up benchmarks by up to 4%**. **Detection:** Flag `Vec::new()` or `vec![]` followed by `.push()` in a loop with a known or estimable bound. Same for `String::with_capacity()` and `HashMap::with_capacity()`. ### Clone abuse and alternatives Cloning heap-allocated data (Vec, String, HashMap) copies all data. The Rust Design Patterns book explicitly identifies "clone to satisfy the borrow checker" as an anti-pattern. Key alternatives: - **`Cow<'a, str>`** for conditional ownership — avoids allocation when no modification is needed - **`Arc<T>`** for shared ownership — clone only increments a reference count - **`clone_from(&b)`** instead of `a = b.clone()` — reuses `a`'s existing heap allocation - **`as_deref()`** for `Option<String>` → `Option<&str>` conversion **Detection:** Flag `.clone()` calls where the value is immediately passed to a function taking `&T`. Clippy's `redundant_clone` lint catches some cases. Flag functions returning `String` that sometimes return the input unchanged (should use `Cow`). ### SmallVec for short-lived small collections `Vec` always heap-allocates when non-empty. For collections that typically hold fewer than 8–16 elements, `SmallVec<[T; N]>` stores them inline on the stack. The rustc compiler uses SmallVec extensively — PRs #50565 and #55383 showed **measurable compilation speed improvements**. **Detection:** Flag `Vec<T>` in struct fields where profiling shows vectors typically contain fewer than 8–16 elements. Particularly valuable for compiler-like workloads, parsers, and intermediate results. ### Arc\<Mutex\<HashMap\>\> contention Coarse-grained locking on a single `Arc<Mutex<HashMap>>` serializes all operations — even reads on different keys block each other. Alternatives ordered by use case: - **`DashMap`**: Sharded `RwLock<HashMap>` with per-bucket locking. Best general-purpose concurrent map. - **`scc::HashMap`**: Fine-grained bucket locks with epoch-based GC. Best for write-heavy workloads. - **`papaya`**: Lock-free reads (RCU-style). Best for read-heavy workloads (no reader-side locking at all). **Detection:** Flag `Arc<Mutex<HashMap<_,_>>>` and `Arc<RwLock<HashMap<_,_>>>` — suggest sharded alternatives. --- ## 4. Serialization choices that multiply latency ### Typed deserialization vs `serde_json::Value` `serde_json::Value` creates a tree of heap-allocated nodes — every string, array, and object is a separate allocation. Typed deserialization into a struct is **1.5–2x faster** (550–710 MB/s vs 300–420 MB/s in json-benchmark). **Detection:** Flag `serde_json::from_str` or `from_reader` with target type `serde_json::Value` when the schema is known. Pattern: deserialization to `Value` followed by field access with string keys like `v["name"]`. ### Zero-copy deserialization with `#[serde(borrow)]` Every `String` field in a `#[derive(Deserialize)]` struct allocates heap memory and copies bytes from the input. Using `&'a str` with `#[serde(borrow)]` borrows directly from the input buffer — **~2x faster** for string-heavy payloads. The `zerovec` crate enables zero-heap-allocation deserialization for vectors. **Detection:** Flag structs with `String` fields used with `Deserialize` where the input lifetime outlives usage. Pattern: `#[derive(Deserialize)]` structs with `String` fields that could be `&str` or `Cow<str>`. **Constraint:** Only works with `from_str`/`from_slice`, not `from_reader`. JSON escape sequences force allocation. ### Binary formats vs JSON for internal APIs JSON is **5–15x slower** than binary formats for serialization. Key benchmarks from `rust_serialization_benchmark`: | Format | Relative speed | Best for | |--------|---------------|----------| | **bincode** | ~40ns ser, ~100ns deser | Fastest general-purpose | | **rkyv** | ~21ns zero-copy deser | Total zero-copy, no parsing step | | **bitcode** | Best combined scores | Newest, excellent compression | | **simd-json** | 2–3x faster than serde_json | JSON-compatible with SIMD | | **postcard** | ~60ns ser, ~180ns deser | Embedded-friendly, compact | | **serde_json** | ~250ns ser, ~500ns deser | Human-readable only | **Detection:** Flag `serde_json` usage in non-user-facing code paths (internal APIs, caches, IPC). Suggest binary formats for machine-to-machine communication. ### `#[serde(untagged)]` enum performance trap Serde's official docs warn that untagged enums try each variant in order, deserializing and backtracking on failure. The input may be parsed **multiple times**. Use `#[serde(tag = "type")]` (internally tagged) or `#[serde(tag = "type", content = "data")]` (adjacently tagged) instead. **Detection:** Flag `#[serde(untagged)]` attribute on enums, especially with many variants in hot deserialization paths. --- ## 5. Concurrency primitives that become bottlenecks ### Unbounded channels: a ticking OOM bomb Unbounded channels (`tokio::sync::mpsc::unbounded_channel`, `crossbeam::channel::unbounded`) have no backpressure. If producers outpace consumers, memory grows without bound until OOM. Tokio issue #4321 documents that even after the spike clears, **memory is never deallocated** from blocks allocated during the spike. Community consensus: "nobody likes unboundedness and most have experienced production outages because of it." **Detection:** Flag any usage of `unbounded_channel()` or `unbounded()`. This is a hard rule — always use bounded channels in production. ### Atomic ordering: SeqCst is almost never needed