dotnet-ci-benchmarking

name	dotnet-ci-benchmarking
category	performance
subcategory	benchmarking
description	Gates CI on perf regressions. Automated threshold alerts, baseline tracking, trend reports.
license	MIT
targets	["*"]
tags	["cicd","dotnet","skill"]
version	0.0.1
author	dotnet-agent-harness
invocable	true
claudecode	{"allowed-tools":["Read","Grep","Glob","Bash","Write","Edit"]}
codexcli	{"short-description":".NET skill guidance for cicd tasks"}
opencode	{"allowed-tools":["Read","Grep","Glob","Bash","Write","Edit"]}
copilot	{}
geminicli	{}
antigravity	{}

dotnet-ci-benchmarking

Continuous benchmarking guidance for detecting performance regressions in CI pipelines. Covers baseline file management with BenchmarkDotNet JSON exporters, GitHub Actions workflows for artifact-based baseline comparison, regression detection patterns with configurable thresholds, and alerting strategies for performance degradation.

Version assumptions: BenchmarkDotNet v0.14+ for JSON export, GitHub Actions runner environment. Examples use actions/upload-artifact@v4 and actions/download-artifact@v4.

Scope

• Baseline file management with BenchmarkDotNet JSON exporters
• GitHub Actions workflows for artifact-based baseline comparison
• Regression detection with configurable thresholds
• Alerting strategies for performance degradation

Out of scope

• BenchmarkDotNet setup and benchmark class design -- see [skill:dotnet-benchmarkdotnet]
• Performance architecture patterns -- see [skill:dotnet-performance-patterns]
• Profiling tools (dotnet-counters, dotnet-trace, dotnet-dump) -- see [skill:dotnet-profiling]
• OpenTelemetry metrics and distributed tracing -- see [skill:dotnet-observability]
• Composable CI/CD workflow design -- see [skill:dotnet-gha-patterns]

Cross-references: [skill:dotnet-benchmarkdotnet] for benchmark class setup and JSON exporter configuration, [skill:dotnet-observability] for correlating benchmark regressions with runtime metrics changes, [skill:dotnet-gha-patterns] for composable workflow patterns (reusable workflows, composite actions, matrix builds).

---

Baseline File Management

BenchmarkDotNet JSON Export

BenchmarkDotNet's JSON exporter produces machine-readable results for automated comparison. Configure the exporter in benchmark classes:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Json;

[JsonExporterAttribute.Full]
[MemoryDiagnoser]
public class CriticalPathBenchmarks
{
    [Benchmark(Baseline = true)]
    public void ProcessOrder() { /* ... */ }

    [Benchmark]
      () {  }
}

```text

Or configure via custom config  all benchmark classes:

```csharp

 BenchmarkDotNet.Configs;
 BenchmarkDotNet.Exporters.Json;
 BenchmarkDotNet.Jobs;
 BenchmarkDotNet.Running;

 config = ManualConfig.Create(DefaultConfig.Instance)
    .AddJob(Job.ShortRun)  
    .AddExporter(JsonExporter.Full)
    .WithArtifactsPath();

BenchmarkSwitcher.FromAssembly((Program).Assembly).Run(, config);

```json





{
  : ,
  : [
    {
      : ,
      : {
        : ,
        : ,
        : ,
        : 
      },
      : {
        : ,
        : ,
        : ,
        : 
      }
    }
  ]
}

```text

Key fields  regression comparison:

| Field                               | Purpose                                         |
| ----------------------------------- | ----------------------------------------------- |
| `Statistics.Mean`                   | ; must update deliberately                       | Small benchmark suites, stable hardware |
| GitHub Actions artifacts         | No repo bloat; automatic retention     | -day  retention; cross-workflow access requires tokens | Large benchmark suites, shared runners  |
| ; cross-repo sharing  | Extra infrastructure; credential management                     | Multi-repo benchmark comparison         |

This skill focuses  the **GitHub Actions artifact** strategy  the . For composable workflow patterns 
reusable actions, see [skill:dotnet-gha-patterns].

---





```yaml

name: Benchmarks

:
  pull_request:
    paths:
      - 
      - 
  workflow_dispatch:

permissions:
  contents: read
  actions: read 

jobs:
  benchmark:
    runs-: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        :
          dotnet-version: 

      - name: Run benchmarks
        run: dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- --exporters json

      - name: Upload benchmark results
        uses: actions/upload-artifact@v4
        :
          name: benchmark-results-${{ github.sha }}
          path: benchmarks/BenchmarkDotNet.Artifacts/results/
          retention-days: 

```text



This workflow downloads the baseline  a previous run  compares against current results:

```yaml

name: Benchmark Regression Check

:
  pull_request:
    paths:
      - 
      - 

permissions:
  contents: read
  actions: read

env:
  BENCHMARK_PROJECT: benchmarks/MyBenchmarks.csproj
  RESULTS_DIR: benchmarks/BenchmarkDotNet.Artifacts/results

jobs:
  benchmark:
    runs-: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        :
          dotnet-version: 

      - name: Download baseline results
        uses: actions/download-artifact@v4
        :
          name: benchmark-baseline
          path: ./baseline-results
        --error: 
        id: download-baseline

      - name: Run benchmarks
        run: dotnet run -c Release --project ${{ env.BENCHMARK_PROJECT }} -- --exporters json

      - name: Compare  baseline
        : steps.download-baseline.outcome == 
        shell: bash
        run: |
           -euo pipefail
          python3 scripts/compare-benchmarks.py \
            --baseline ./baseline-results \
            --current  \
            --threshold  \
            --output benchmark-comparison.md

      - name: Upload current results   baseline
        : github. == 
        uses: actions/upload-artifact@v4
        :
          name: benchmark-baseline
          path: ${{ env.RESULTS_DIR }}/
          retention-days: 
          overwrite: 

      - name: Upload comparison report
        : steps.download-baseline.outcome == 
        uses: actions/upload-artifact@v4
        :
          name: benchmark-comparison-${{ github.sha }}
          path: benchmark-comparison.md
          retention-days: 

```markdown

**Key design decisions:**

- `--error: `  baseline download handles first-run (no baseline exists yet)
- Baseline  only updated  `main` branch merges to prevent PR branches  polluting the baseline
- `overwrite: ` replaces the previous baseline artifact

For converting these inline workflows  reusable `workflow_call` patterns, see [skill:dotnet-gha-patterns].

---





Compare current benchmark results against baseline  percentage thresholds. A regression  flagged  the current
mean exceeds the baseline mean  more than the configured threshold:

```python




 = {}
     = json.load(f)
         bm  data.(, []):
            name = bm[]
            benchmarks[name] = {
                : bm[][],
                : bm[][],
                : bm[][],
                : bm.(, {}).(, ),
            }
      = load_benchmarks(baseline_dir)
    current = load_benchmarks(current_dir)
    regressions = []

     name, curr  current.items():
         name   baseline:
              
         = baseline[name]
         [] == :
              

        time_change_pct = ((curr[] - []) / []) * 
        alloc_change = curr[] - []

         time_change_pct > threshold_pct:
            regressions.append({
                : name,
                : [],
                : curr[],
                : time_change_pct,
                : alloc_change,
            })

     regressions

 __name__ == :
    import argparse

    parser = argparse.ArgumentParser(description=)
    parser.add_argument(, =True, help=)
    parser.add_argument(, =True, help=)
    parser.add_argument(, type=, =,
                        help=)
    parser.add_argument(, =, help=)
     = parser.parse_args()

    regressions = compare(.baseline, .current, .threshold)

    ; small regressions are signal                |
| GitHub Actions shared runners | %              | Shared runners introduce % variance  noisy neighbors |
| Self-hosted runners           | %               | More stable than shared, but still monitor variance          |

**Calibrate thresholds empirically:** Run the same benchmark suite  times  your CI environment without code
changes. The maximum observed variance sets your noise floor.  ==  && github.event_name == 
  uses: actions/github-script@v7
  :
    script: |
       fs = require();
       body = fs.readFileSync(, );
       github.rest.issues.createComment({
        owner: context.repo.owner,
        repo: context.repo.repo,
        issue_number: context.issue.number,
        body: body
      });

```text



Exit  non-zero status  the comparison script to fail the GitHub Actions job. This prevents merging PRs that
introduce performance regressions:

```yaml

- name: Check  regressions
  : steps.download-baseline.outcome == 
  shell: bash
  run: |
     -euo pipefail
    python3 scripts/compare-benchmarks.py \
      --baseline ./baseline-results \
      --current  \
      --threshold 
    

```text

For  status checks  branch protection integration  benchmark gates, see [skill:dotnet-gha-patterns].



For -term trend analysis beyond single-PR comparison, upload results to a persistent store  track metrics over
time:

| Approach                           | Tool                                          | Complexity                            |
| ---------------------------------- | --------------------------------------------- | ------------------------------------- |
| GitHub Actions artifacts           | Built-, -day retention                    | Low -- artifact download/upload only  |
| GitHub Pages  benchmark-action | `benchmark-action/github-action-benchmark@v1` | Medium -- auto-generates trend charts |
| External time-series DB            | InfluxDB, Prometheus + Grafana                | High -- full observability stack      |

The simplest approach  most projects  the artifact-based baseline comparison shown   skill. Graduate to trend
tracking  you need historical regression analysis across many releases.

---





Full benchmark runs take + minutes. Use `Job.ShortRun`  CI to reduce iteration counts  retaining regression
detection capability:

```csharp

 BenchmarkDotNet.Configs;
 BenchmarkDotNet.Jobs;

   : 
{
    
    {
        AddJob(Job.ShortRun
            .WithWarmupCount()
            .WithIterationCount()
            .WithInvocationCount());

        AddExporter(BenchmarkDotNet.Exporters.Json.JsonExporter.Full);
    }
}

```json

Apply conditionally based  environment:

```csharp

 config = Environment.GetEnvironmentVariable()   
    ?  CiConfig()
    : DefaultConfig.Instance;

BenchmarkRunner.Run<CriticalPathBenchmarks>(config);

```text



Run only critical-path benchmarks  CI to reduce pipeline duration:

```bash


dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- \
  --filter *Critical* --exporters json

```bash

```csharp

[]
[]
[]
  
{
    []
     {  }
}

[]
[]
  
{
    []
     {  }
}

```text

Run `Critical` benchmarks  every PR; run `Extended` benchmarks  a nightly schedule.



```yaml

name: {{ github.run_number }}
          path: benchmarks/BenchmarkDotNet.Artifacts/results/
          retention-days: 

```text

For scheduled workflow patterns  matrix builds across TFMs, see [skill:dotnet-gha-patterns].

---





**Primary approach:** Use Serena symbol operations  efficient code navigation:

 **Find definitions**: `serena_find_symbol` instead of text search
 **Understand structure**: `serena_get_symbols_overview`   organization
 **Track references**: `serena_find_referencing_symbols`  impact analysis
 **Precise edits**: `serena_replace_symbol_body`  clean modifications

**When to use Serena vs traditional tools:**
- ✅ **Use Serena**: Navigation, refactoring, dependency analysis, precise edits
- ✅ **Use Read/Grep**: Reading full files, pattern matching, simple text operations
- ✅ **Fallback**: If Serena unavailable, traditional tools work fine

**Example workflow:**
```text

Read: src/Services/OrderService.cs
Grep: 


serena_find_symbol: 
serena_get_symbols_overview: 
```


 **Use `Job.ShortRun`  CI,  `Job.Default`** --  benchmark jobs run many iterations  statistical
   precision, taking + minutes per benchmark . CI pipelines need faster feedback  `ShortRun` ( warmup, 
   iteration).
 **Set threshold above measured noise floor** -- shared CI runners introduce % timing variance  noisy
   neighbors. A % threshold  shared runners produces  positives. Calibrate  running the same code multiple
   times  measuring variance.
 **Use allocation changes  hard gates** -- allocation counts are deterministic  unaffected  runner noise. A
   zero-to-nonzero allocation change  always a real regression, unlike timing variations.
 **Only update baselines  main branch** --  PR branches can update the baseline, a regression  one PR becomes
   the  baseline, masking it  subsequent comparisons.
 **Always  ` -euo pipefail`  bash steps** -- without `pipefail`, a regression detection script that exits
   non-