grpc-resilience

name	grpc-resilience
description	gRPC resilience patterns — retry policies, hedging, deadlines, cancellation, and transient fault handling for .NET gRPC clients. Use when implementing fault-tolerant gRPC communication.

gRPC Resilience Patterns

Retry Policy

Automatic retry for failed gRPC calls. Built into Grpc.Net.Client (v2.36.0+).

Configuration

var defaultMethodConfig = new MethodConfig
{
    Names = { MethodName.Default },
    RetryPolicy = new RetryPolicy
    {
        MaxAttempts = 5,
        InitialBackoff = TimeSpan.FromSeconds(1),
        MaxBackoff = TimeSpan.FromSeconds(5),
        BackoffMultiplier = 1.5,
        RetryableStatusCodes = { StatusCode.Unavailable }
    }
};

var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
{
    ServiceConfig = new ServiceConfig { MethodConfigs = { defaultMethodConfig } }
});

Retry Options

Option	Description
MaxAttempts	Total attempts including the original (min 2, capped by GrpcChannelOptions.MaxRetryAttempts default 5)
InitialBackoff	Initial delay between retries (must be > 0)
MaxBackoff	Upper limit for exponential backoff growth
BackoffMultiplier	Multiplier applied to backoff after each attempt
RetryableStatusCodes	Status codes that trigger a retry (at least one required)

Common Retryable Status Codes

• StatusCode.Unavailable — service temporarily down
• StatusCode.DeadlineExceeded — timeout (use cautiously)
• StatusCode.Aborted — transaction conflict
• StatusCode.ResourceExhausted — rate limited (with backoff)

When Retries Are Valid

Calls are retried when ALL conditions are met:

1. Status code matches RetryableStatusCodes
2. Previous attempts < MaxAttempts
3. Call is NOT committed (no response headers received, buffer not exceeded)
4. Deadline not exceeded

Retry Detection

The server can detect retries via grpc-previous-rpc-attempts metadata:

public override Task<Response> MyMethod(Request request, ServerCallContext context)
{
    var retryAttempt = context.RequestHeaders
        .FirstOrDefault(h => h.Key == "grpc-previous-rpc-attempts")?.Value;
    // null = first attempt, "1" = first retry, etc.
}

---

Hedging Policy

Aggressively send multiple copies of a call without waiting for a response. First successful result wins.

var defaultMethodConfig = new MethodConfig
{
    Names = { MethodName.Default },
    HedgingPolicy = new HedgingPolicy
    {
        MaxAttempts = 5,
        HedgingDelay = TimeSpan.FromMilliseconds(500),
        NonFatalStatusCodes = { StatusCode.Unavailable }
    }
};

Hedging vs Retry

Aspect	Retry	Hedging
Execution	Sequential (wait for failure)	Parallel (send eagerly)
Latency	Higher (waits for each failure)	Lower (first success wins)
Server load	Lower	Higher (multiple calls)
Idempotency	Recommended	Required (multiple executions)
Combinable	Not with hedging	Not with retry

Important: Hedging is only safe for idempotent operations. A hedging policy CANNOT be combined with a retry policy on the same method.

---

Deadlines

Always set deadlines on client calls to prevent indefinitely hanging RPCs.

// Option 1: Deadline on individual call
var deadline = DateTime.UtcNow.AddSeconds(5);
var reply = await client.GetItemAsync(
    new GetItemRequest { Id = "123" },
    deadline: deadline);

// Option 2: Default deadline via CallOptions
var options = new CallOptions(deadline: DateTime.UtcNow.AddSeconds(10));
var reply = await client.GetItemAsync(request, options);

// Option 3: Default deadline via interceptor (recommended for consistency)

Server-Side Deadline Handling

public override async Task<Response> LongOperation(
    Request request, ServerCallContext context)
{
    // Check if deadline already exceeded
    if (context.CancellationToken.IsCancellationRequested)
    {
        throw new RpcException(new Status(StatusCode.Cancelled, "Deadline exceeded"));
    }

    // Pass cancellation token to downstream operations
    var result = await _service.ProcessAsync(request, context.CancellationToken);
    return new Response { Data = result };
}

---

Cancellation

Client-Side

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5));
try
{
    var reply = await client.GetItemAsync(
        new GetItemRequest { Id = "123" },
        cancellationToken: cts.Token);
}
catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled)
{
    // Call was cancelled
}

Server-Side

public override async Task StreamData(
    Request request,
    IServerStreamWriter<DataChunk> responseStream,
    ServerCallContext context)
{
    while (!context.CancellationToken.IsCancellationRequested)
    {
        var chunk = await GetNextChunkAsync(context.CancellationToken);
        await responseStream.WriteAsync(chunk, context.CancellationToken);
    }
}

---

Streaming Resilience

Retry Limitations with Streaming

• Server streaming / bidirectional: Retries stop after first response message is received
• Client streaming / bidirectional: Retries stop if outgoing messages exceed the buffer
• Buffer size is controlled by MaxRetryBufferSize and MaxRetryBufferPerCallSize

Manual Reconnection Pattern

For streaming calls that cannot be auto-retried:

async Task StreamWithReconnect(CancellationToken ct)
{
    while (!ct.IsCancellationRequested)
    {
        try
        {
            using var call = client.ServerStreamMethod(new Request());
            await foreach (var msg in call.ResponseStream.ReadAllAsync(ct))
            {
                ProcessMessage(msg);
            }
            break; // Stream completed normally
        }
        catch (RpcException ex) when (ex.StatusCode == StatusCode.Unavailable)
        {
            await Task.Delay(TimeSpan.FromSeconds(2), ct);
            // Retry the stream
        }
    }
}

---

Transient Fault Handling Checklist

1. Configure retry policies for idempotent unary calls
2. Set deadlines on ALL client calls
3. Use CancellationToken in all async operations
4. Handle RpcException with status code inspection
5. Implement manual reconnection for long-lived streams
6. Consider hedging only for latency-critical idempotent operations
7. Monitor grpc-previous-rpc-attempts metadata on the server
8. Log transient failures at Warning level, permanent failures at Error