performance-review

name	performance-review
description	Review Rust codebases for performance anti-patterns, with emphasis on async runtimes, client-server networking, memory allocation, serialization, and concurrency. Use when asked to audit, review, or optimize Rust code for performance.

Rust Performance Review Skill

You are a Rust performance auditor. Your job is to scan Rust source code and project configuration for known performance anti-patterns and report findings with concrete fix suggestions.

How to use this skill

1. Start by reading Cargo.toml and Cargo.lock to understand dependencies, the build profile, target platform, and whether the project uses async (tokio/async-std), networking (hyper/tonic/axum/actix/reqwest), or serialization (serde, serde_json, bincode, etc.).

2. Check Tier 1 (project config) issues first — these are free wins that require no code changes: LTO, codegen-units, target-cpu, panic mode, global allocator on musl.

3. Scan source files for Tier 2 (code pattern) issues using the reference below. Prioritize findings by estimated impact: async executor blocking and missing TCP_NODELAY are nearly always the highest-impact fixes.

4. Report findings grouped by severity (critical / warning / suggestion), with the file path, line reference, the problematic pattern, why it matters, and the recommended fix. Keep explanations terse — link to the relevant section of this reference for details.

Severity classification

• Critical: Will cause production incidents under load (executor blocking, unbounded channels, connection-per-request, missing buffered I/O).
• Warning: Measurable performance loss, 2x+ degradation in affected path (wrong mutex type, SeqCst everywhere, Arc<Mutex>, serde_json for internal APIs, Vec without with_capacity in hot loops, clone abuse).
• Suggestion: Optimization opportunity, 5–20% improvement potential (LTO, PGO, binary serialization, zero-copy deserialization, SmallVec, vectored I/O, custom allocator).

Key detection patterns (quick reference)

Blocking in async: std::fs::, std::thread::sleep, std::net::, block_on inside async fn.

Networking: Missing set_nodelay(true), raw TcpStream without BufReader/BufWriter, TcpStream::connect in request handlers (no pool), to_socket_addrs() in async code.

Memory: Vec::new() + loop .push() without with_capacity, .clone() passed to &T parameter, format!() inside loops, Arc<Mutex<HashMap>>.

Serialization: serde_json::Value with known schema, #[serde(untagged)], String fields in Deserialize structs that could borrow.

Concurrency: unbounded_channel(), Ordering::SeqCst, adjacent atomics without cache padding, tokio::sync::Mutex with no .await inside lock scope.

Config: Missing lto, codegen-units > 1, no target-cpu=native, musl without #[global_allocator].

Full reference

The sections below contain the complete catalog of ~50 anti-patterns with detection strategies, impact analysis, fixes, and real-world case studies. Consult them for details when reporting findings.

---

1. Async runtime pitfalls that silently kill throughput

Blocking the executor

The single most common Rust async performance bug is running blocking operations on tokio worker threads. Every call to std::thread::sleep, std::fs::read, synchronous DNS resolution, or CPU-heavy computation (bcrypt, compression) prevents the cooperative scheduler from switching tasks. Alice Ryhl (Tokio maintainer) states that async code should never spend more than 10–100 microseconds without reaching an .await. With 4 runtime threads and 100ms blocking per request, maximum throughput collapses to 40 req/s — with no errors or panics, just silent latency degradation.

Detection signals: Grep for std::thread::sleep, std::fs::, std::net::TcpStream, std::io::Read, std::io::Write, and block_on inside async fn or async blocks. Check for synchronous database drivers (e.g., diesel without spawn_blocking). The tokio-console tool measures task poll durations, and the hud crate provides zero-instrumentation blocking detection.

Fix: Use tokio::task::spawn_blocking for blocking I/O, tokio::fs for file operations, tokio::time::sleep for delays, and offload CPU-heavy work to rayon via a oneshot channel bridge.

Sequential .await chains destroying concurrency

Rust futures are lazy — they do nothing until polled. Consecutive let a = fetch_a().await; let b = fetch_b().await; runs operations serially even when they're independent, making total time equal to the sum rather than the maximum.

Detection: Look for multiple consecutive let x = something().await; lines where operations don't depend on each other's results. An AST check can identify independent await expressions in sequence.

Fix: Use tokio::join! for a fixed set of futures, JoinSet for dynamic collections, or FuturesUnordered for stream-based processing.

Wrong mutex type selection

Using tokio::sync::Mutex when std::sync::Mutex suffices incurs ~2x overhead. The async mutex is only needed when the lock must be held across .await points. Conversely, holding a std::sync::MutexGuard across an .await causes compile errors on multi-threaded runtimes (the guard is !Send) or deadlocks on single-threaded runtimes.

Detection: Flag tokio::sync::Mutex where no .await occurs between lock acquisition and release. Flag std::sync::Mutex guards whose scope spans an .await point. Pattern: .lock() followed by .await without an intervening drop() or scope boundary.

Large futures and stack pressure

Async functions compile into state machines where every variable held across an .await becomes a field in the future's struct. Large stack arrays, deeply nested async calls, or many local variables can create futures of tens to hundreds of kilobytes — the Fuchsia team observed 400KB single futures. Since Tokio 1.41.0, futures larger than 16KB (release) passed to tokio::spawn are automatically boxed, but the allocation cost remains.

Detection: Grep for large array declarations [T; N] where N is large inside async functions. Use std::mem::size_of_val(&future) to measure. tokio-console reports size.bytes per task.

Cancellation safety in select!

When a tokio::select! branch completes, other branches' futures are dropped. Operations like read_exact, write_all, and Mutex::lock are not cancellation-safe — partial progress is silently lost, causing data corruption or starvation.

Detection: Flag tokio::select! or futures::select! in loops containing read_exact, write_all, BufReader::read_line, or other non-cancellation-safe methods. The Tokio docs annotate each method's cancellation safety status.

---

2. Networking configuration mistakes with outsized latency impact

TCP_NODELAY: the number-one networking fix

By default, TCP uses Nagle's algorithm, which buffers small writes and delays sending until a full segment accumulates or an ACK arrives. For RPC and interactive protocols, this adds 40–200ms of latency per request. Axum issue #2521 documented massive latency regressions because TCP_NODELAY wasn't set by default. The community consensus is emphatic: "It's always TCP_NODELAY."

Detection: Grep for every TcpStream::connect and listener.accept() call and verify that set_nodelay(true) follows. Check hyper/axum/tonic configuration for TCP_NODELAY settings.

Missing buffered I/O on network streams

Every read() or write() on a raw TcpStream triggers a system call. The ScyllaDB Rust driver team discovered this was their #1 performance bottleneck — their driver issued at least one syscall per query, causing 2x CPU usage compared to their C++ driver. Wrapping streams with BufReader/BufWriter is a drop-in fix that batches operations into 8KB chunks by default.

Detection: Flag TcpStream used directly for read/write without BufReader/BufWriter wrapping. Check flamegraphs for excessive time in sendmsg/recvmsg syscalls.

Connection pool exhaustion and DNS blocking

Without connection pooling, each request opens a new TCP connection plus TLS handshake plus authentication — 50–150ms overhead per connection. Even with pools, exhaustion occurs when pools are undersized or connections are held across slow operations. Standard library DNS resolution (ToSocketAddrs::to_socket_addrs()) is synchronous and will block the async runtime.

Detection: Flag TcpStream::connect in request handlers (should use a pool). Flag to_socket_addrs() in async code. Flag pool.get() where the resulting connection spans unrelated .await calls. Verify reqwest clients configure pool_idle_timeout and pool_max_idle_per_host.

HTTP/2 flow control and window sizing

Default HTTP/2 flow control windows (typically 64KB) throttle large message transfers. Tonic issue #569 documented that transferring 0.1GB took over a minute with default settings because flow control required many round-trips for window updates. Setting http2_initial_stream_window_size, http2_initial_connection_window_size, and enabling http2_adaptive_window resolves this.

Detection: Flag tonic/hyper servers transferring large payloads without custom HTTP/2 window configuration. Check for SETTINGS_MAX_CONCURRENT_STREAMS configuration to prevent resource exhaustion.

---

3. Memory allocation patterns that compound under load

Heap allocation in hot paths

Each heap allocation involves acquiring a global lock, non-trivial bookkeeping, and possibly a system call. The Rust Performance Book states that reducing allocations by 10 per million instructions can yield measurable improvements (~1%). Common offenders include Vec::new(), String::new(), Box::new(), and format!() inside loops.

Detection: Flag allocating constructors (Vec::new(), String::new(), format!(), Box::new()) inside loop, for, and while blocks. Use DHAT profiler to identify hot allocation sites.

Fix: Reuse buffers across calls by passing &mut String or &mut Vec<T> parameters. Use write! to a reusable buffer instead of format!(). Pre-allocate with Vec::with_capacity() when the size is known.

Vec reallocations from missing with_capacity

Vec's growth strategy (0→4→8→16→32→64...) means pushing 1000 items one-by-one causes ~10 reallocations, each copying all existing elements. A rustc PR showed that adjusting Vec's initial growth reduced allocations by 10%+ and sped up benchmarks by up to 4%.

Detection: Flag Vec::new() or vec![] followed by .push() in a loop with a known or estimable bound. Same for String::with_capacity() and HashMap::with_capacity().

Clone abuse and alternatives

Cloning heap-allocated data (Vec, String, HashMap) copies all data. The Rust Design Patterns book explicitly identifies "clone to satisfy the borrow checker" as an anti-pattern. Key alternatives:

• Cow<'a, str> for conditional ownership — avoids allocation when no modification is needed
• Arc<T> for shared ownership — clone only increments a reference count
• clone_from(&b) instead of a = b.clone() — reuses a's existing heap allocation
• as_deref() for Option<String> → Option<&str> conversion

Detection: Flag .clone() calls where the value is immediately passed to a function taking &T. Clippy's redundant_clone lint catches some cases. Flag functions returning String that sometimes return the input unchanged (should use Cow).

SmallVec for short-lived small collections

Vec always heap-allocates when non-empty. For collections that typically hold fewer than 8–16 elements, SmallVec<[T; N]> stores them inline on the stack. The rustc compiler uses SmallVec extensively — PRs #50565 and #55383 showed measurable compilation speed improvements.

Detection: Flag Vec<T> in struct fields where profiling shows vectors typically contain fewer than 8–16 elements. Particularly valuable for compiler-like workloads, parsers, and intermediate results.

Arc<Mutex<HashMap>> contention

Coarse-grained locking on a single Arc<Mutex<HashMap>> serializes all operations — even reads on different keys block each other. Alternatives ordered by use case:

• DashMap: Sharded RwLock<HashMap> with per-bucket locking. Best general-purpose concurrent map.
• scc::HashMap: Fine-grained bucket locks with epoch-based GC. Best for write-heavy workloads.
• papaya: Lock-free reads (RCU-style). Best for read-heavy workloads (no reader-side locking at all).

Detection: Flag Arc<Mutex<HashMap<_,_>>> and Arc<RwLock<HashMap<_,_>>> — suggest sharded alternatives.

---

4. Serialization choices that multiply latency

Typed deserialization vs serde_json::Value

serde_json::Value creates a tree of heap-allocated nodes — every string, array, and object is a separate allocation. Typed deserialization into a struct is 1.5–2x faster (550–710 MB/s vs 300–420 MB/s in json-benchmark).

Detection: Flag serde_json::from_str or from_reader with target type serde_json::Value when the schema is known. Pattern: deserialization to Value followed by field access with string keys like v["name"].

Zero-copy deserialization with #[serde(borrow)]

Every String field in a #[derive(Deserialize)] struct allocates heap memory and copies bytes from the input. Using &'a str with #[serde(borrow)] borrows directly from the input buffer — ~2x faster for string-heavy payloads. The zerovec crate enables zero-heap-allocation deserialization for vectors.

Detection: Flag structs with String fields used with Deserialize where the input lifetime outlives usage. Pattern: #[derive(Deserialize)] structs with String fields that could be &str or Cow<str>.

Constraint: Only works with from_str/from_slice, not from_reader. JSON escape sequences force allocation.

Binary formats vs JSON for internal APIs

JSON is 5–15x slower than binary formats for serialization. Key benchmarks from rust_serialization_benchmark:

Format	Relative speed	Best for
bincode	~40ns ser, ~100ns deser	Fastest general-purpose
rkyv	~21ns zero-copy deser	Total zero-copy, no parsing step
bitcode	Best combined scores	Newest, excellent compression
simd-json	2–3x faster than serde_json	JSON-compatible with SIMD
postcard	~60ns ser, ~180ns deser	Embedded-friendly, compact
serde_json	~250ns ser, ~500ns deser	Human-readable only

Detection: Flag serde_json usage in non-user-facing code paths (internal APIs, caches, IPC). Suggest binary formats for machine-to-machine communication.

#[serde(untagged)] enum performance trap

Serde's official docs warn that untagged enums try each variant in order, deserializing and backtracking on failure. The input may be parsed multiple times. Use #[serde(tag = "type")] (internally tagged) or #[serde(tag = "type", content = "data")] (adjacently tagged) instead.

Detection: Flag #[serde(untagged)] attribute on enums, especially with many variants in hot deserialization paths.

---

5. Concurrency primitives that become bottlenecks

Unbounded channels: a ticking OOM bomb

Unbounded channels (tokio::sync::mpsc::unbounded_channel, crossbeam::channel::unbounded) have no backpressure. If producers outpace consumers, memory grows without bound until OOM. Tokio issue #4321 documents that even after the spike clears, memory is never deallocated from blocks allocated during the spike. Community consensus: "nobody likes unboundedness and most have experienced production outages because of it."

Detection: Flag any usage of unbounded_channel() or unbounded(). This is a hard rule — always use bounded channels in production.

Atomic ordering: SeqCst is almost never needed