corvus-bytes-to-bytes

name

corvus-bytes-to-bytes

description

Eliminate hand-rolled POCO record<->document string seams — types/paths that materialize a managed string (or List<string>/Dictionary) between a bytes SOURCE (a parsed UTF-8 body, a DB column, a directory/HTTP response) and a bytes SINK (a serialized JSON document, a generated model, a SecurityTagSet, a DB write). The fix recipe is bytes-native default + opt-in string extensibility, plus a decision procedure for telling a genuine string leaf from work you are avoiding. USE FOR: deciding whether a string on a hot/warm path is justified; converting a bytes->string->bytes u-turn to bytes-to-bytes; reviewing your own diff for hidden string materializations before committing; building an identity/tag set from a UTF-8 source. DO NOT USE FOR: the mechanics of building a generated model from spans without a closure (use corvus-builder-context-threading), raw pooling/stackalloc patterns (use corvus-buffer-and-pooling), constructing models from native values (use corvus-typed-model-construction).

Bytes-to-bytes: killing record<->document string seams

A record<->document string seam is the most expensive allocation anti-pattern in this codebase and the one most often re-introduced. It is a value that arrives as bytes, is materialized into a managed string (or List<string> / Dictionary<string,…> / a per-row POCO), and is then written back out as bytes — a u-turn through the managed heap. Both ends are bytes; the string is pure overhead, plus a closure if a non-static builder lambda is involved.

The cost is real and measured: a per-row row-security scan was 25.78 KB -> 1.56 KB (0.06x) once the filter walked the persisted UTF-8 instead of SecurityTagSet.ToList() per row; a directory grantee projection was 5.88 KB -> 2.01 KB (0.34x) once value/label flowed as spans instead of GetString.

The genuine-leaf proof (run this BEFORE you write a string)

The words "genuine leaf", "marginal", "admin-rare", "low-frequency", "pragmatic", "good enough", "consistency win", "too fragile", "edge case", "one X's worth" are red flags: they are usually a justification reached for first, to license skipping the bytes mechanism. Before any of them excuses a managed string on a warm/hot path, write a two-ended trace and check both ends:

	Source end	Destination end
String IS the leaf only if	a string-typed external API: ClaimsPrincipal claims, the Novell LDAP client (LdapAttribute.StringValue), a BsonValue.AsString (the driver pre-materialized it)	a string-typed sink: an HttpRequestMessage URI, an LDAP filter, an HTTP Authorization header, a string-keyed store (store.GetAsync(string)), a human-facing audit/error message

If EITHER end is bytes/spans and a documented mechanism exists, it is NOT a leaf — it is work you are avoiding. A "constructed" value (e.g. first + " " + last, a prefix + dimension key) is never a leaf: assemble it into a pooled/stack UTF-8 buffer, not a string. The mechanism always exists:

Need	Mechanism	Skill
Build a tag set from UTF-8	SecurityTagSet.Build + IdentityBuilder.Add(span)	this skill
Build a key prefix + dimension	stackalloc/ArrayPool + Encoding.UTF8.GetBytes(prefix, key)	corvus-buffer-and-pooling
Write UTF-8 into a generated model, no closure	T.Build<TContext> / CreateBuilder<TContext>	corvus-builder-context-threading
Compare UTF-8 vs a fixed string	pre-encode the string to u8/byte[] once, SequenceEqual	this skill
Re-transcode a holder's bytes	Utf8JsonWriter -> TagSet.CopyFromJsonArray	corvus-mutable-documents

When the excuse is "too fragile" / "edge case" (a self-imposed invariant)

Sometimes the string (or the List + per-item concat that builds it) isn't defended as a leaf but as too risky to remove — "the sort has a '-' vs '.' edge case needing a careful comparer", "I'd have to reproduce the exact output bytes", "it's only one row's worth". That difficulty is almost always self-imposed: it comes from preserving the existing code's incidental output shape (an entry/element order, a container layout, an insertion order), not from the task itself. Before you skip it: (1) name the exact invariant making it fragile; (2) read the consumer — the reader, the sink, the comparer — and check whether it actually requires that invariant. An order-independent reader (finds entries by name/key, not position), a sink that re-normalises/re-canonicalises, or a content-addressed-by-hash artifact does not. When no consumer requires it, the invariant is incidental — drop it and the clean low-alloc form falls out. Example: WorkflowPackage.PackPooled sorts sources by key in a pooled scratch array and emits a fixed bucket order (workflow, sources, metadata), writing each entry name as UTF-8 directly ("sources/"u8 + key + ".json"u8, length back-patched) — instead of reproducing the old full-name sort, which removed the List + per-source name string outright (PackCanonicalPackage 0.73 -> 0.49 KB, scales per source). Treating incidental output shape as a contract is the anchoring-on-existing-code failure (deriving the replacement from what the old code happened to do) applied to behaviour instead of style.

Opaque tokens are a carrier seam too ("store-minted, so it stays a string" is a rationalization)

An opaque pagination/continuation token round-trips between two UTF-8 ends — emitted into a JSON response, carried back in the next JSON request (a CTJ JsonString) — so "it is store-minted, not domain data, so it stays a string" is a genuine-leaf rationalization: both ends are bytes. Encode it bytes-native with System.Buffers.Text.Base64Url (GetEncodedLength + EncodeToUtf8 straight into the destination; GetMaxDecodedLength + DecodeFromUtf8 from the request's pageToken.GetUtf8String().Span) — no EncodeToString/DecodeFromChars char detour, and no per-part ToString()/concat/GetBytes (assemble the key into a stackalloc/ArrayPool UTF-8 buffer with a separator byte). The token must survive into the response, but owned ≠ GC: pool it in a disposable carrier rather than minting a string (the only necessarily-GC form). The cross-root request→store JsonString is bridged with JsonString.From(...) (free rewrap). The lifetime that governs emitting it is the deferred-body rule — see corvus-ctj-handler-implementation.

The fix recipe — bytes-native default + opt-in string

Mirror IdentityBuilder.Add(ReadOnlySpan<byte> key, ReadOnlySpan<byte> value) (the fast path) vs Add(ReadOnlySpan<byte> key, string value) (the opt-in for a value a deployment genuinely computed through a string API). The bytes path is the path; the string path is the explicit, documented exception — never the default.

// The bytes-native seam every adapter/handler builds an identity through. The value span is the
// unescaped UTF-8 the source already holds (reader.GetUtf8String().Span); no managed string per tag.
SecurityTagSet identity = SecurityTagSet.Build(
    in state,
    static (ref IdentityBuilder builder, in TState s) =>
    {
        builder.Add("sys:tenant"u8, s.TenantSpan);   // span path
        builder.Add("sys:sub"u8, s.SubSpan);
    });

A deferred holder (SecurityTagSet, TagSet) is the bytes form. Read it as ((JsonElement)x).GetUtf8String() (a ref struct; cannot cross an await) or .TakeOwnership(out byte[]? rented) for owned ReadOnlyMemory<byte> that can. Transcode to a string ONLY at the genuine leaf (e.g. Encoding.UTF8.GetString(value.Span) for a string-keyed store key).

Dual constructor for an extension-point contract

When a type is a deployment-authored extension point (e.g. ResolvedPrincipal, returned by an IDirectoryIdentityMapper), give it BOTH constructors — span for the built-in fast path, string for mapper ergonomics — and decode on demand for string consumers:

public readonly struct ResolvedPrincipal
{
    private readonly ReadOnlyMemory<byte> value;   // owned UTF-8

    // Span ctor — the built-in adapters' bytes-to-bytes fast path (copies the transient span to owned).
    public ResolvedPrincipal(GranteeKind kind, ReadOnlySpan<byte> value, ReadOnlySpan<byte> label, bool hasLabel, SecurityTagSet identity) { /* value.ToArray() */ }
    // String ctor — the ergonomic path a deployment mapper (or LDAP, a genuine string leaf) uses.
    public ResolvedPrincipal(GranteeKind kind, string value, string? label, SecurityTagSet identity) { /* Encoding.UTF8.GetBytes(value) */ }

    public ReadOnlyMemory<byte> ValueMemory => this.value;                 // server: bytes-to-bytes into the response
    public string Value => Encoding.UTF8.GetString(this.value.Span);       // CLI/tests: decode on demand
}

Constructed value -> pooled buffer (never a string)

// A "first last" display name has no single source span — assemble it into a REUSED pooled buffer
// (rented once outside the row loop, grown on demand), not Encoding.UTF8.GetString(...) + string concat.
int needed = first.Length + 1 + last.Length;
if (labelBuffer is null || labelBuffer.Length < needed)
{
    if (labelBuffer is not null) { ArrayPool<byte>.Shared.Return(labelBuffer); }
    labelBuffer = ArrayPool<byte>.Shared.Rent(needed);
}
first.CopyTo(labelBuffer);
labelBuffer[first.Length] = (byte)' ';
last.CopyTo(labelBuffer.AsSpan(first.Length + 1));
ReadOnlySpan<byte> labelSpan = labelBuffer.AsSpan(0, needed);   // the consumer copies it before the next row reuses the buffer

Walking a holder's UTF-8 instead of materializing it

When a path must evaluate over a tag set (not just copy it), parse the persisted UTF-8 once into a pooled scratch + slice table (SecurityTagSpanSort.Parse), compare on spans, and pre-encode the fixed side once. The string evaluator becomes a thin adapter that delegates via FromTags, so the existing tests lock the bytes evaluator's semantics. See SecurityRule.EvaluateAll(in SecurityTagSet, in Utf8ClaimSet) and SecurityRuleEvaluation.cs. Ordinal UTF-8 SequenceEqual IS ordinal string equality for the same code points.

Pre-commit self-audit (mandatory — see copilot-instructions.md pre-commit gate)

Before committing, scan your own diff and report each:

• New managed string / List<string> / Dictionary on a path where bytes are available? -> apply the proof; fix or flag.
• New non-static builder lambda (a closure) where a static + TContext form exists? -> corvus-builder-context-threading.
• Reflection-based dispatch where a virtual/span seam exists? -> remove it.
• Any work deferred / skipped / relocated? -> surface it under Decisions & deferrals + a task, never a buried comment.
• Did a fix MOVE a cost (a transcode/alloc) elsewhere rather than remove it? -> state the before/after file:line.

Prove every warm path with a BenchmarkDotNet [MemoryDiagnoser] benchmark (baseline old vs new) — see SecurityFilterScanBenchmarks / GranteeProjectionBenchmarks for the shape. "Admin-rare" is not a licence to allocate.

Cross-References

• corvus-builder-context-threading — write UTF-8 spans into a generated model with no closure (the Build<TContext> form).
• corvus-buffer-and-pooling — the stackalloc/ArrayPool/thread-local pooling the span paths rent from.
• corvus-typed-model-construction — constructing generated models from native values (the non-span common case).
• ref-struct-delegates — why the build callbacks are named ref struct delegates, not Func<>/Action<>.
• corvus-parsed-documents-and-memory — GetUtf8String() / TakeOwnership / the deferred-holder memory model.