sglang-runtime-context

name

sglang-runtime-context

description

How SGLang's runtime configuration and process-global state are organized (RuntimeContext tiers, publish + namespace config bags, the pristine ServerArgs seed, override entry points, resource/stream/buffer leases, per-forward flags), the CI guardrails that enforce the design, and the idioms for developing and testing against it. Load this before touching server_args, model overrides, module-level state, or per-forward state in sglang.

SGLang runtime-context architecture

One container owns process-static runtime state: sglang.srt.runtime_context.RuntimeContext (a process singleton reached via get_context()). Everything below is a tier on it.

Tier	Accessor	Holds	Lifecycle
raw config seed	get_server_args()	the published pristine ServerArgs (resolved-at-startup record; kept for debugging, dumps, per-runner fork copies)	published at process entry; re-publish is last-publish-wins (in-process tokenizer build, multi-Engine) and re-projects the bags; read-only
resolved config	get_exec() get_memory() get_schedule() get_model() get_spec() get_serving() get_observability() get_disagg() get_lora() get_mm() get_device()	namespace config bags — the single source of truth for resolved config; leaves are real attributes (dynamo-traceable)	projected from server_args at publish; mutated only via get_context().override
runtime flags	get_flags()	state that is not a pure function of config: capture (cuda-graph lifecycle), moe (ACTIVE backends, swappable), dp (DP-attention runtime flags)	materialized at subsystem init; groups offer override() for tests
resources	get_resources(), get_stream(name), get_buffer(name, factory)	process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers	lazy; cleared by reset_context()
per-forward	get_forward()	forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch)	contextvar-backed; scoped(**kw) restores on exit; new threads see defaults
parallel	get_parallel()	dual: live topology (tp/pp/moe/attn sizes, ranks, groups — @property, read-through) plus parallel config-bag leaves via __getattr__	live: after dist init; config leaves: after publish

reset_context() (unit-test teardown) drops the published config and installs fresh flags/resources/forward tiers.

Config: publish + namespace bags

ServerArgs is a pristine seed. Business code never reads it for decisions — resolved configuration lives in the namespace bags.

• Every publishing process entry calls publish(server_args, role=...) (run_scheduler_process, the Ray SchedulerActor, the DP controller, tokenizer, encoder, weight-cache daemon, launcher, ...). The one deliberate exception is the detokenizer: its processes never publish and read only the raw config handed to their constructors — code that can run detokenizer-side must not use the namespace accessors. publish snapshots the resolved field values into the config bags; the accessors (get_exec() etc.) fail closed before it runs. role records which process type published, and keys per-role namespace enforcement: SGLANG_ROLE_NAMESPACES=record audits which namespaces each role's process actually reads (per-pair persisted via SGLANG_ROLE_NAMESPACES_OUT; reads inside torch.compile-traced code are NOT observed — audit with compilation disabled before restricting a role), and =enforce fails closed on bag reads outside the role's ROLE_NAMESPACE_SETS entry (None = full tree; only audited roles are restricted).
• Bag membership is metadata on the dataclass: every ServerArgs field carries NS("path") (e.g. NS("exec.moe")); coverage is linted two-way (test_server_args_namespaces.py, test_runtime_context_config_bags.py).
• Reading config: get_<ns>()[.sub].field — e.g. get_exec().moe.moe_a2a_backend, get_schedule().max_running_requests. Bag leaves are plain instance attributes, safe inside torch.compile-traced code.
• Mutating config after publish: the ONLY entry point is get_context().override(source, **fields). It writes the bag leaves in place (namespace readers see the new value) and records provenance in the overrides log. There is no write-through to the ServerArgs instance — it stays pristine. There is no in-place mutation entry on the instance at all: it is read-only after resolution.
• Late launcher-stage resolution (pre-publish): a few rules cannot run inside __post_init__ — LoRA normalization, and the auto-parser detection that needs a tokenizer/chat-template load. They are resolution, not mutation, and they write in place via arg_groups.overrides.declare_late_resolution(server_args, source, **fields), which refuses the published instance. In place is the point: every holder of that object must see the resolved value — the HTTP server, the multi-tokenizer workers it is serialized for, the schedulers it forks. Returning a variant here is a bug: the launcher rebinds its local and everyone else keeps the unresolved object.

Why a bag override cannot stand in for the last two. The bags are projected at publish from the instance's fields, so anything the runtime must read has to be on the instance before publish — an override afterwards puts instance and bags back out of agreement, and whole-object readers (ModelConfig.from_server_args, build_load_config, MMEncoder's own self.server_args.X) never see it. And bags do not cross a process boundary: a child publishes from the object it receives and re-projects its own bags, so a parent-side override is lost. Values that feed construction before any bag exists (group init reads server_args.tp_size) have no bag to override at all.

• Nested publishes: a construction step that must publish a private copy wraps itself in get_context().preserve_config() — the enclosing lifecycle, including its post-publish overrides, is value-snapshotted and reinstated on exit. The draft build no longer needs it: per-runner values are constructor arguments now.

Reads that legitimately stay on a ServerArgs instance

• Per-runner (fork) fields — fields the draft-worker deepcopy rewrites (attention_backend, prefill/decode_attention_backend, speculative_draft_attention_backend, skip_tokenizer_init, context_length, load_format, json_model_override_args, kv_cache_dtype): each runner's copy is authoritative for that runner, so runner code reads self.server_args.X, and resolved per-runner values live as runner attributes (model_runner.kv_cache_dtype_str is the pattern — threaded to consumers as constructor args, never backfilled onto shared objects).
• Per-instance boundaries — the tokenizer-manager family, everything under entrypoints/, and the tokenizer-process multimodal processors read self.server_args: several Engines can share one process, and the process-global bags are last-publish-wins across engines. base_gpu_id also differs per worker (the encode-server DP workers each specialize their own copy), so no process-global value can stand in for it — BaseMultimodalProcessor._fast_image_processor_device is the shape to copy.
• Whole-object passes (f(server_args) handing the instance along) keep the supplied-instance contract; don't rewrite the parameter reads to bag reads unless the field is runtime-mutated (see the elastic-EP ep_size case in eplb/expert_location.py).

get_parallel(): config leaves vs live topology

Config leaves (nccl_port, enable_dp_attention, dp_size, ep_size, dwdp_size, ...) resolve through the parallel bag; live topology (tp_size, attn_tp_group, ranks) are @property and win on name collisions. Five topology sizes are live-shadowed (tp/pp/dcp/attn_cp/moe_dp_size): a config-intent read of those must stay on server_args.X — the live property always wins on the accessor. Fail-loud is narrower: before dist init, any live size/group read raises; after it, only the DCP group is optional (_DCP exists only when dcp_size > 1; attn-CP and moe-DP always install, as size-1 aliases if unused). ParallelContext.__getattr__ is deliberately dynamo-traceable (no object.__getattribute__); gate helpers like enable_moe_dense_fully_dp() run inside compiled model forwards (test_parallel_config_leaves_trace_under_torch_compile pins this).

Mid-resolution reads (inside the pipeline only)

Resolution itself still runs in __post_init__: handlers and hooks read the in-flight state through resolved_view(server_args) / self._resolved(), fields are read-only during resolution, and declarations materialize once at the very end of __post_init__ (gate order, last writer wins) — then publish snapshots the resolved values into the bags. resolved_view is pipeline-internal (server_args.py / arg_groups/, plus helpers the pipeline itself invokes mid-resolution, e.g. adaptive_spec_params); do not introduce new out-of-pipeline call sites.

Adding a model-specific config adjustment

Never assign server_args fields from model code. Declare instead (sglang/srt/arg_groups/overrides.py):

• Constant per-arch values → MODEL_OVERRIDES["MyArchForCausalLM"] = {...}.
• Derived values → @register_model_override("MyArchForCausalLM") returning a dict; the callable receives pristine server_args + hf_config and must not write.
• Normalization that must see earlier declarations → a post-process pass invoked via run_post_process_pass at its slot (reads a view, returns a declaration dict).
• Values only knowable at weight-load time → declare_load_time_override(source, {...}) — validates the whitelist, then routes through get_context().override (bag-only; the declaration lands on the published bags, not on any ServerArgs instance). Scope caveat for draft models: only a draft build that publishes a private copy under preserve_config discards its declarations with the scope. Draft loads that skip publish share the process bags, so their declarations land process-wide — declares reachable from a draft load must be draft-safe (guard or same-value).

Declarable fields form a whitelist: Arg(..., resolvable=True) in the ServerArgs dataclass. A declaration against a non-whitelisted field fails at its slot.

Load-time vs resolution-time (critical)

__post_init__ runs in the launcher process before any model/platform import. Logic that consults an extensible registry (e.g. out-of-tree platforms registering attention backends in init_backend(), which runs at model_runner import) must stay at load time (ModelRunner init), writing through get_context().override(). Before moving any load-time logic into resolution, verify everything it reads is already complete at construction time.

Runtime flags (get_flags())

For state that init-time code derives and runtime code reads — parsed enums, platform probes, swappable ACTIVE values. Not for config mirrors (read the bag leaf instead).

• Groups are typed dataclasses on Flags (capture / moe / dp): typo-safe writes, transactional test-only override(**kw) context manager.
• flags.moe is materialized by initialize_moe_config(server_args) at scheduler init; accessors (get_moe_a2a_backend etc.) are thin shims with lazy defaults. The speculative contexts (speculative_moe_backend_context) swap the ACTIVE leaves around draft forwards.
• flags.dp is materialized by initialize_dp_attention; is_dp_attention_enabled() is a shim over flags.dp.enabled.
• Adding a leaf: declare the dataclass field with a default equal to the pre-init behavior, materialize it at the owning subsystem's init, keep any public accessor as a shim.

Resources (get_resources())

Named slots + two keyed-lazy registries:

• get_stream(name) — get-or-create a named CUDA side stream; set_stream(name, stream) installs explicitly. Name leases by subsystem ROLE: all model alternate streams share "alt"; the offloader's copy stream is "offload"; DP-TBO comm is "dp_tbo_comm"; LoRA side stream is "lora_side". Two call sites may share a name only if their work belongs on one stream — sharing across roles serializes intended overlap.
• get_buffer(name, factory) — get-or-create a named persistent buffer. Grow-only or per-device semantics manage their resources.buffers entries directly (see tokenspeed / SM120 split / Marlin workspace). Buffer names are per-backend today; do not silently share.
• Singletons with manager semantics (EP dispatcher buffers, EPLB recorder/metadata, graph memory pool) keep their owning accessors/classes as facades; only the state lives in a resources entry. Preserve exact semantics in the shim: lazy defaults (the EPLB recorder defaults to a Noop instance, not None), publish-once asserts, event-reuse contracts.
• Stream/buffer creation is a driver call — it must happen outside cuda-graph capture; keep lease points at init/warmup time.

Per-forward flags (get_forward())

Contextvar-backed; a new thread sees the defaults; scoped(**kw) is the regular write path (transactional, restores on exit and on exception); set(name, value) exists for legacy sticky setters (is_extend_in_batch is intentionally sticky within a thread). Use this tier for anything set-per-forward and read-within-forward. Before adding cross-thread state here, prove the readers' thread affinity: contextvars do NOT propagate to already- running or newly spawned threads. Note TBO ("two-batch overlap") interleaves ubatches on ONE thread — do not design for TBO threads that don't exist.

Testing idioms

• Force a code path by overriding causes, not effects: compose get_context().override_server_args(**fields) (publishes a fresh dummy-boundary ServerArgs carrying the overrides AND projects the bags — with-scoped, or install()/restore() + addCleanup for fixture-lifetime use) + get_<ns>().override(...) (scoped override of one bag's own leaves) + get_parallel().override(...) (live topology) + get_flags().<group>.override(...) + get_forward().scoped(...). All are scoped and transactional. Tests control execution through the context — do not hand-build and publish config objects.
• Never monkeypatch import bindings (module.get_x = lambda: ...) and never fake a config source with a SimpleNamespace stand-in: production reads the published bags, so a faked accessor silently stops intercepting after any reader migration. Publish for real (override_server_args(...)), then adjust bag leaves with the scoped bag override where the constructed ServerArgs cannot carry the value (e.g. get_device().override(device="meta")).
• Mocked runners/managers still need the per-runner instance attributes the code under test reads (kv_cache_dtype_str, server_args for whole-object passes) — set them explicitly on the mock; MagicMock(spec=...) raises on attributes that only exist post-__init__, which is the fastest way to find a missed stub.
• reset_context() in teardown when a test publishes outside a scoped override.
• ServerArgs(model_path="dummy") early-returns __post_init__ (no materialization, no strict guard) — fine for lightweight fixtures.
• Run changed test files per-file (own process), the way CI does: a monolithic local pytest run lets a context published by an earlier file mask a missing-publish bug in a later one.

Guardrails (these fail CI; what to do when they fire)

1. Strict mutation guard (always on): bare server_args.x = ... after resolution raises unconditionally in ServerArgs.__setattr__ — this is the guarantee that no writer can desync the bags, so there is no writer ratchet any more. Change resolved config with get_context().override, build a per-runner config with server_args.derive. Projected bags are sealed the same way (leaf assignment raises).
2. Mutation ratchet (test_server_args_mutation_ratchet.py, exact pin 0 over the whole package minus the pipeline / multimodal_gen): textual scan for assignment forms. Never raise the baseline.
3. Derive contract (test_server_args_derive.py): deriving leaves the receiver intact, a published config still refuses assignment, and deriving does not publish. Rerouting a writer to the bags means flipping all its readers in the same commit (no transitional dual-write).
4. Legacy-accessor ratchet (test_legacy_global_ratchet.py): get_global_server_args call sites must not grow — new code uses runtime_context.get_server_args() (and business decisions should read the bags).
5. Module-state ratchet (test_module_state_ratchet.py): global statements in the flag-owning layers are pinned by name. A new module-level runtime global belongs on a flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
6. Namespace coverage (test_server_args_namespaces.py, test_runtime_context_config_bags.py): every ServerArgs field carries NS(...) metadata and the projected bags must cover the fields exactly (two-way).

Never module-skip a test "until the migration settles" — seed the context instead (the deferral ratchet that once pinned this is retired; the rule stands).

Hard-won pitfalls (check these before/while refactoring)

• Moving code drops first-line guards: early returns (if self.is_draft_worker: return) are the easiest thing to lose when relocating a method body. Every draft is built under a preserved publish of its own config: the scheduler makes the copy with draft_server_args_copy() (seeded from the resolved config, so load-time overrides carry) and publishes it around the worker factory, and build_draft_tp_worker() nests the same shape for dflash / dspark. The publish ends when construction does — anything the draft reads later (alloc_memory_pool, init_attention_backends, cuda-graph capture) is back on the target's bags.
• Registry-completeness timing: a gate that consults an extensible list is only correct after the registrars ran (platform init_backend() at module import). See "load-time vs resolution-time".
• Late function-scope imports shadow module names for the WHOLE function (UnboundLocalError at earlier lines). Audit moves with AST, not grep.
• Lease names are per-role, not per-API-shape (the offloader-vs-"alt" lesson).
• Storage matrix for state read inside torch.compile-traced model code (piecewise cuda graph compiles the whole model forward): contextvars are untraceable (hard error); dict-slot values are guarded per value — for a per-forward int that is one recompile per distinct size, straight into the recompile limit; class/instance attributes are the only compile-friendly form (attribute-source ints get automatic-dynamic after the first size change). Bools (≤2 values) are tolerable in any form — see ForwardFlags._GRAPH_VISIBLE. Config-bag leaves are real instance attributes for exactly this reason, and ParallelContext.__getattr__ must stay free of object.__getattribute__ (dynamo graph-breaks on it). Before moving such state, prove its readers sit outside compile coverage; a piecewise-prefill boot of a small model is the fast check (recompile storms show as torch._dynamo hit config.recompile_limit during the compile pass).
• Engine-booting e2e tests are the only coverage for launcher-path code; a child crash kills the process tree and pytest dies silently — run with PYTHONUNBUFFERED=1 and read child logs.
• CI arms SGLANG_ENABLE_ASYNC_ASSERT=1 (device-side torch._assert_async probes, e.g. KV-cache OOB): a fired device assert kills the tree with no Python traceback, and the same bug is silent corruption locally with the flag off. Arm it when reproducing CI crashes.

Where to read the code

Key source files: python/sglang/srt/runtime_context.py (the container, every tier, publish, _ConfigBag, preserve_config, override_server_args), python/sglang/srt/arg_groups/overrides.py (override registry, passes, declare_load_time_override), python/sglang/srt/server_args.py (NS metadata, Arg(..., resolvable=True), __setattr__ strict guard), and the guardrail tests under test/registered/unit/ (test_server_args_mutation_ratchet.py, test_server_args_writer_ratchet.py, test_legacy_global_ratchet.py, test_module_state_ratchet.py, test_server_args_namespaces.py, test_runtime_context.py — the last one doubles as executable documentation of every tier's semantics).