blazing-fast-dotnet

# Blazing Fast .NET Performance > [!WARNING] > Much of the advice here might not age well: as .NET evolves, old hacks and custom knobs may become obsolete or even detrimental. Apply only after careful consideration and testing. Revisit periodically to ensure continued relevance. This skill focuses on optimizing .NET applications for maximum performance, providing tips, techniques, and best practices to achieve blazing-fast execution times. It targets .NET 10.0 and 11.0. Older versions have not been tested. **Contents** - [Profiling and Performance Analysis](#profiling-and-performance-analysis) - [dotnet-counters](#dotnet-counters) - [dotnet-trace](#dotnet-trace) - [Low-level profiling with Linux `perf`](#low-level-profiling-with-linux-perf) - [Popular low-level problems](#popular-low-level-problems) - [Inspecting JIT codegen](#inspecting-jit-codegen) - [Best practices for high-performance C# code](#best-practices-for-high-performance-c-code) - [Quick tips](#quick-tips) - [Strings and globalization](#strings-and-globalization) - [Contention on shared state](#contention-on-shared-state) - [False sharing](#false-sharing) - [Safe vectorization (SIMD or SWAR)](#safe-vectorization-simd-or-swar) - [Generic specialization via static abstract members](#generic-specialization-via-static-abstract-members) - [ASP.NET Core](#aspnet-core) - [Magic knobs to Tune .NET Performance](#magic-knobs-to-tune-net-performance) - [NativeAOT-specific tuning](#nativeaot-specific-tuning) - [General notes](#general-notes) # Profiling and Performance Analysis Start cheap and go deeper only when needed: counters (what kind of bottleneck?) → traces (where?) → `perf` and JIT disassembly. Install the tools with: ```sh dotnet tool install -g dotnet-counters dotnet tool install -g dotnet-trace dotnet tool install -g dotnet-symbol ``` ## dotnet-counters [dotnet-counters](https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-counters) is a low-overhead, first-level triage tool. Run it while the app is under load: ```sh dotnet-counters monitor -p <pid> # System.Runtime counters by default dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting,System.Net.Http dotnet-counters collect -p <pid> --format csv -o counters.csv # save for post-processing ``` See the `System.Runtime` [built-in metrics](https://learn.microsoft.com/dotnet/core/diagnostics/built-in-metrics-runtime) (.NET 9+) for what each counter means. ## dotnet-trace [dotnet-trace](https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-trace) collects EventPipe traces on any OS without root or a native profiler (managed frames only): ```sh dotnet-trace collect -p <pid> --duration 00:00:00:10 # default profiles: dotnet-common + dotnet-sampled-thread-time dotnet-trace collect -p <pid> --profile gc-verbose # GCs + sampled allocations: which types/stacks allocate dotnet-trace collect -p <pid> --clrevents contention # lock contention events with stacks dotnet-trace collect -- ./MyApp arg1 # trace from startup dotnet-trace list-profiles dotnet-trace report trace.nettrace topN -n 20 # top methods (exclusive) as plain text dotnet-trace report trace.nettrace topN -n 20 --inclusive dotnet-trace convert trace.nettrace --format Speedscope # open in https://www.speedscope.app ``` For more tips on using `dotnet-trace`, refer to the [dotnet-trace collection skill](https://raw.githubusercontent.com/dotnet/skills/refs/heads/main/plugins/dotnet-diag/skills/dotnet-trace-collect/SKILL.md). ## Low-level profiling with Linux `perf` The instructions below are mostly for CoreCLR/JIT scenarios on Linux with the native `perf` tool. NativeAOT/R2R scenarios should presumably work as is, without `perf inject`. Typically, the following environment variables need to be set in the context of the running application: ```sh DOTNET_PerfMapEnabled=1 DOTNET_PerfMapStubGranularity=3 DOTNET_JitFramed=1 DOTNET_EnableWriteXorExecute=0 ``` What to profile depends on the scenario. Typically it's steady-state performance, so wait until the process is properly warmed up (e.g. 1 minute), then attach and collect traces: ```sh perf record -k 1 -g -F 3999 -p $PROCESS_PID -o perf.data -- sleep 5 ``` Next, symbolize the collected JIT frames using `perf inject`: ```sh perf inject --input perf.data --jit --output perfjit.data ``` Then use the resulting `perfjit.data` file as needed, e.g. to generate flame graphs or annotated asm: ```sh # annotated asm & report: perf report --input perfjit.data --no-children --percent-limit 1 --stdio > perf_report.txt perf annotate --stdio2 -i perfjit.data --percent-limit 1 -M intel > perf.asm # Generate flamegraph (svg) git clone --depth 1 --quiet https://github.com/brendangregg/FlameGraph || true perf script -i perfjit.data | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > flamegraph.svg ``` Now `perf.asm` contains the annotated assembly output for the hottest methods in your application. Keep in mind that sample attribution is imprecise ("skid"): the cost is often reported on an instruction shortly after the one that actually stalls (e.g. right after a slow load or division), so inspect the neighboring instructions too. In case of missing symbols you may need to use `dotnet-symbol` to download the necessary debug symbols for .dll/.dylib/.so: ```sh dotnet-symbol --symbols $PUBLISH_FOLDER/*.dll dotnet-symbol --symbols $PUBLISH_FOLDER/*.so ``` All other `perf`-related techniques (e.g. c2c, branch profiling, cache misses) can be applied similarly, depending on the performance aspects you want to analyze. For simple functions, create a self-contained console application that calls the function in an infinite loop. For BenchmarkDotNet, use the in-process toolchain (`-i` or `[InProcess]`) with a long iteration duration and a very large iteration count so it is clear which process to attach `perf` to. ## Popular low-level problems - Cache contention and false sharing in multi-threaded code. Fix with per-thread data and padding (see [False sharing](#false-sharing)). - Bounds checks not being elided by the JIT. - Store-to-load forwarding stalls, e.g. writing a struct field by field and reading it back with a wider load. - Bad data locality causing cache misses. - Code alignment noise. Loop or branch placement, Intel JCC erratum - not something we can control, just to keep in mind when analyzing performance variations. - GC's alignment is 8 bytes (64-bit), which might not be suitable for SIMD optimizations (e.g. AVX512 might penalize every load/store if misaligned since the cache line is 64 bytes). ## Inspecting JIT codegen These are diagnostic switches (not tuning knobs) and work with the shipped Release runtime: ```sh DOTNET_JitDisasm="Test" # method list: 'Test', 'Program:Test', '*Test*', 'Program:*' (space-separated) DOTNET_JitDisasmSummary=1 # list every jitted method with its tier DOTNET_JitStdOutFile=/tmp/jit.txt # write JIT output to a file instead of stdout DOTNET_JitDisasmDiffable=1 # stable output (no addresses) for before/after diffs DOTNET_JitDisasmOnlyOptimized=1 # show only Tier-1/FullOpts code ``` ^ might require `MethodImplOptions.NoInlining` on the method to ensure it gets JIT-compiled and not inlined away. > [!TIP] > Set them only for the app process, not for `dotnet run`, which would also dump the SDK's own methods. - A method is compiled several times as it tiers up. Judge the last version: header `; Tier1 code` (or `; Tier1-OSR code` / `; FullOpts code`), ideally with `; optimized using Dynamic PGO`. `; Tier0 code` and `; Instrumented Tier0 code` are unoptimized. - BCL methods precompiled with ReadyToRun show up only once they get hot and are re-jitted at Tier1. - NativeAOT: pass the same options to ILC, e.g. `<IlcArg Include="--codegenopt:JitDisasm=Test" />` and `<IlcArg Include="--codegenopt:JitStdOutFile=jit.txt" />`. # Best practices for high-performance C# code > [!IMPORTANT] > Only safe C# code is considered in these best practices. Unsafe code is highly discouraged and not covered here. ## Quick tips - Use [source-generated System.Text.Json](https://learn.microsoft.com/dotnet/standard/serialization/system-text-json/source-generation): pass generated `JsonTypeInfo<T>` to avoid reflection-based metadata discovery and enable supported serialization fast paths. Preserve existing serialization options and behavior. - Use [`[LoggerMessage]`](https://learn.microsoft.com/dotnet/core/extensions/logging/source-generation) source-generated logging to avoid template parsing, boxing, and temporary allocations. Still guard expensive argument computation with `IsEnabled`. - Use [`[GeneratedRegex]`](https://learn.microsoft.com/dotnet/standard/base-types/regular-expression-source-generators) for compile-time-known patterns instead of runtime pattern compilation. Preserve options, culture, and match timeouts. - Prefer optimized bulk APIs over handwritten loops: `Span.CopyTo`, `Clear`, `Fill`, `SequenceEqual`, and `IndexOf` already have specialized implementations, including SIMD where appropriate. - Use RVA-backed spans `static ReadOnlySpan<T> RVA => new T[CNS]` for all kinds of immutable data (only primitives, e.g., `long`, `byte`, `char`) - the data will be embedded into metadata, and the JIT will be able to perform constant folding on it. - Prefer static lambdas and state-passing overloads in hot paths to avoid capturing closures. - Parse and format UTF-8 directly: numeric `TryParse(ReadOnlySpan<byte>, ...)` and `TryFormat(Span<byte>, ...)` avoid intermediate strings and UTF-8/UTF-16 transcoding. Preserve culture and formatting semantics. - Cache `SearchValues<char>` / `SearchValues<byte>` for repeated searches against fixed sets; use `SearchValues<string>` for multi-substring searches. Construction has a cost, so do not create these per call. - Consider `FrozenDictionary` / `FrozenSet` for build-once, read-many data. Their construction cost must be amortized over enough lookups; preserve the intended comparer. - Use dictionary alternate lookups from spans: `dictionary.GetAlternateLookup<ReadOnlySpan<char>>()` avoids allocating temporary string keys when the comparer supports it. Use `TryGetAlternateLookup` when comparer support is not known. - Implement `IEquatable<T>` and a good `GetHashCode` for struct dictionary/set keys. This can avoid expensive default equality paths, boxing, and collision-heavy hashes. - Try [`TensorPrimitives`](https://learn.microsoft.com/dotnet/api/system.numerics.tensors.tensorprimitives) before hand-vectorizing numeric operations. Its span-based implementations handle SIMD and tails; prefer it over manual SIMD loops or LINQ for things like finding min/max, sums, etc. - Prefer [`[LibraryImport]`](https://learn.microsoft.com/dotnet/standard/native-interop/pinvoke-source-generation) over `[DllImport]` for supported signatures to generate marshalling at build time; both `DllImport` and `LibraryImport` require `AllowUnsafeBlocks=true` under unsafe-v2, but if interop is not avoidable, `LibraryImport` is generally preferred. - Use `static readonly` flags for genuinely immutable process-wide configuration. Optimized JIT code can fold their values and eliminate disabled branches. - Cache fields in locals in hot loops (`var items = _items;`). The JIT often can't do it for you: calls and stores through references may modify the field, and the memory model restricts hoisting reads. Locals also enable bounds check elimination for arrays/spans. - Consider `Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping` for UTF-8 JSON-only responses ([documentation](https://learn.microsoft.com/dotnet/standard/serialization/system-text-json/character-encoding#serialize-all-characters)). Never embed its raw output in HTML or `<script>`; any custom encoder disables STJ's generated serialization fast path, so benchmark rather than assuming the gains add up. - For most bit/byte manipulations, prefer the built-in `BitOperations`, `BinaryPrimitives`, `BitConverter` or static numeric helpers such as `uint.Log2`, `uint.PopCount`, `uint.TrailingZeroCount`, etc. - Span/ReadOnlySpan have a lot of useful extension methods via `MemoryExtensions` such as `SequenceEqual`, `IndexOf`, `IndexOfAnyExcept`, `LastIndexOf`, `StartsWith`, `CommonPrefixLength`, and a lot more - prefer these APIs over manual loops or vectorization. - Don't pin benchmarks to a single core - doing so might significantly affect internal VM heuristics (e.g. tiered compilation timing). - `AggressiveOptimization` is usually a deoptimization. - Overusing `AggressiveInlining` may hurt performance (e.g. by exhausting the inliner's time budget or increasing the number of tracked local variables). - Don't put too much data into TLS (Thread-Local Storage) - excessive use can lead to increased memory usage and cache pressure, reducing overall performance (there can be a lot of threads, each with its own copy). - Consider calling `.AsSpan()` on `List<T>` or other collections that support it to avoid unnecessary overhead from bound and covariance checks, and to enable span-based algorithms directly on the collection's underlying memory. ## Strings and globalization - Prefer ordinal operations. Culture-sensitive comparisons go through ICU, are many times slower than ordinal ones, and are often a correctness bug anyway. - Culture-sensitive by default: `string.Compare`, `CompareTo`, `StartsWith(string)`, `EndsWith(string)`, `IndexOf(string)`, `LastIndexOf(string)`, `ToUpper()`, `ToLower()`, and default string sorting (`Array.Sort`, `List<T>.Sort`, `OrderBy`, `SortedDictionary`/`SortedSet` with the default comparer). Pass `StringComparison.Ordinal`/`OrdinalIgnoreCase` or `StringComparer.Ordinal`; analyzers CA1307, CA1309 and CA1310 find these calls. - Already ordinal: `==`, `Equals`, `Contains(string)`, `Replace(string, string)`, `IndexOf(char)`, `Split`, `GetHashCode`, and the default `Dictionary`/`HashSet` comparer. - Case-insensitive equality: `string.Equals(a, b, StringComparison.OrdinalIgnoreCase)` instead of `a.ToLower() == b.ToLower()` (two allocations plus culture rules). For ASCII data, `System.Text.Ascii` has vectorized `EqualsIgnoreCase`, `ToUpper`/`ToLower` (span to span) and `IsValid`. - `Dictionary<string, T>` / `HashSet<string>` use fast non-randomized hashing only with the default comparer, `StringComparer.Ordinal` or `StringComparer.OrdinalIgnoreCase` (falling back to randomized hashing only on heavy collisions). Any other comparer loses it. - Stay in UTF-8 when input and output are UTF-8: search, parse and format `ReadOnlySpan<byte>` directly (`u8` literals, `SearchValues<byte>`, `TryParse`/`TryFormat` on bytes, `Utf8.TryWrite`) and convert to `string` only when needed. - If the app needs no culture data, `<InvariantGlobalization>true</InvariantGlobalization>` removes the ICU dependency (faster startup, less memory) and makes culture-sensitive string operations ordinal. This changes behavior, so verify the output. ## Contention on shared state Writes to one shared location from many threads (`Interlocked.Increment` on a global counter, a shared "total" field, etc.) serialize all of them on a single cache line, so throughput can get *worse* as cores are added. Split the state, update only the local part, and combine on read. Two options: **1. Per-core stripes** - index an array by `Thread.GetCurrentProcessorId()` (cheap: cached and periodically refreshed). The id is only a hint: threads migrate, and ids can exceed `ProcessorCount` (hence the modulo), so keep `Interlocked` on the stripe - it's almost always uncontended. Fixed memory and allocation-free reads; works with any number of threads (thread pool, request handlers). Stripes must be padded, see [False sharing](#false-sharing). ```cs sealed class StripedCounter { private readonly PaddedLong[] _stripes = new PaddedLong[Environment.ProcessorCount]; public void Increment() { PaddedLong[] stripes = _stripes; Interlocked.Increment(ref stripes[(uint)Thread.GetCurrentProcessorId() % (uint)stripes.Length].Value); } // Not an atomic snapshot, which is fine for statistics. public long Sum() { PaddedLong[] stripes = _stripes; long sum = 0; for (int i = 0; i < stripes.Length; i++) sum += Volatile.Read(ref stripes[i].Value); return sum; } } ``` **2. `ThreadLocal<T>` with `trackAllValues: true`** - each thread updates its own slot with no atomics at all; sum `Values` on read. `Values` allocates a list and includes values of threads that have already exited, and every thread that ever touched it keeps a slot, so this suits a bounded set of long-lived threads (e.g. `Parallel` workers) where writes vastly outnumber reads. Dispose the `ThreadLocal<T>` when done. ```cs private readonly ThreadLocal<long> _count = new(trackAllValues: true); void OnItem() => _count.Value++; long Total() => _count.Values.Sum(); ``` If the work structure allows it, aggregating into plain locals and combining once per task/partition (`Parallel.For` with `localInit`/`localFinally`) is cheaper than both. ### False sharing Threads writing *different* variables still contend if those variables share a cache line (64 bytes on most x64/arm64 CPUs, 128 bytes on Apple M-series): every write invalidates the line in the other cores' caches. The naive striped counter is the classic case: ```cs // 8 longs per 64-byte line: the per-core slots look independent, but cores keep stealing the line from each other. long[] stripes = new long[Environment.ProcessorCount]; Interlocked.Increment(ref stripes[(uint)Thread.GetCurrentProcessorId() % (uint)stripes.Length]); ```