dynamo-sglang-bump

name	dynamo-sglang-bump
description	Bump Dynamo's SGLang backend to a new SGLang version. Sets up a clean dynamo main checkout, target SGLang tag, fresh venv, then walks every launch script and fixes API breakage as it surfaces. Use when the user asks to upgrade/bump SGLang in Dynamo (e.g. "update dynamo for sglang 0.5.X").
user-invocable	true

Dynamo SGLang Version Bump

End-to-end recipe for upgrading the dynamo.sglang backend to a new SGLang release.

The upgrade is bottom-up empirical: spin up the canonical environment, then run each launch script in examples/backends/sglang/launch/ and fix breakage as it appears. Don't try to predict the diff from release notes — let the scripts surface real failures.

Canonical Paths

Path	Purpose
/ephemeral/dynamo	Dynamo checkout. Always start on main and pull.
/ephemeral/sglang	SGLang upstream checkout. Checkout the target tag (vX.Y.Z).
~/aiperf	Benchmark client (rarely needed for a bump).

If these paths don't exist on the box, ask the user before cloning to a new location — they may have a different layout.

Step 1: Confirm Inputs

Before touching anything, confirm with the user:

• Target SGLang version (e.g. 0.5.9, mapped to tag v0.5.9).

• Runtime container tags. Always ask — these don't always match the pip version 1:1 (.post1 suffixes, RC tags, CUDA variants). Need:

  • The CUDA-default runtime tag, e.g. v0.5.11-runtime
  • The CU130 runtime tag, e.g. v0.5.11-cu130-runtime Both go into container/context.yaml, container/rendered.Dockerfile, container/compliance/README.md, and any container/templates/sglang_runtime.Dockerfile references. Never guess. Ask explicitly: "What lmsysorg/sglang runtime image tags should I bump to for the default and cu130 variants?"

  Verify the tags exist on Docker Hub before applying — docker buildx imagetools inspect lmsysorg/sglang:v<ver>-runtime is the cheapest check. SGLang ships base images and runtime images via two separate GitHub Actions workflows ("Release Docker Images" and "Release Docker Runtime Images"), and the runtime workflow has historically failed/been delayed for some releases (e.g. v0.5.11's runtime build initially died on apt-mirror flake during apt-get update). If the runtime tag doesn't exist yet:

  • Ask the user whether to (a) wait for upstream to re-run the runtime workflow (gh run list --repo sgl-project/sglang --workflow "Release Docker Runtime Images"), (b) push to a maintainer with permission to dispatch it (workflow_dispatch accepts a version input matching X.Y.Z), or (c) defer the container tag bump to a follow-up PR and ship the pip bump alone. Option (c) matches the precedent at commit 26af597bf85 ("chore: SGLang base image refresh ...") and is the right call when there's no eta on the runtime images.
  • Whichever path the user picks, mention it explicitly in the bump PR body so reviewers don't think the container files were forgotten.

• Branch name. Default convention: idhanani/sgl-to-<ver>-and-cleanups (matches the 0.5.9 precedent on ishan/sgl-to-0.5.9-and-cleanups).

• Linear ticket if there is one, for the PR description.

Step 2: Reset Dynamo to main

cd /ephemeral/dynamo
git status                         # must be clean; if not, ask the user
git checkout main
git pull --ff-only origin main
git checkout -b <branch-name>

Refuse to proceed if the working tree is dirty — those are the user's in-progress changes.

Step 3: Checkout SGLang at Target Tag

cd /ephemeral/sglang
git fetch origin --tags
git checkout v<target-version>     # e.g. v0.5.9
git status                         # verify detached HEAD at the tag

If the tag is missing, run git fetch origin --tags --force and retry. Don't silently fall back to a different ref.

Step 4: Fresh venv

A fresh venv is not optional for this skill — old SGLang artifacts in site-packages cause confusing import-time errors that look like Dynamo bugs.

cd /ephemeral/dynamo
deactivate 2>/dev/null || true
rm -rf .venv-sgl-<version>          # name it after the target version
uv venv .venv-sgl-<version> --python 3.12
source .venv-sgl-<version>/bin/activate

Verify isolation:

which python
python -c "import sglang" 2>&1 | head -1   # expected: ModuleNotFoundError

Step 5: Install SGLang (Local Editable)

Install from the /ephemeral/sglang checkout, not from PyPI — this lets you grep into the live SGLang source while debugging. Always include [diffusion] — image/video/dllm launch scripts pull in diffusers, imageio, imageio-ffmpeg, moviepy etc. via that extra. Without it you'll bounce off ModuleNotFoundError: imageio / diffusers two scripts in.

cd /ephemeral/sglang && uv pip install -e "python[diffusion]"
python -c "import sglang; print(sglang.__version__)"

The reported version must match the target. If pip resolves a different one, look for a requirements*.txt constraint pinning it elsewhere.

Step 6: Build Dynamo Bindings + Install

A fresh venv has neither maturin nor the nixl Python bindings — install both first or maturin develop will command not found and dynamo workers will fail at import with ImportError: NIXL Python bindings must be installed.

uv pip install maturin nixl
cd /ephemeral/dynamo/lib/bindings/python && maturin develop --uv
cd /ephemeral/dynamo && uv pip install -e .

Sanity check the rebuilt Rust exports — the kvstats symbol on dynamo.prometheus_names was missing during the 0.5.9 bump until the bindings were rebuilt:

python -c "from dynamo.prometheus_names import kvstats; print('ok')"

Step 7: Known Environment Gotchas

Encode these as preflight env vars for the test session:

• CuDNN mismatch (SGLANG_DISABLE_CUDNN_CHECK=1) PyTorch ships an older CuDNN than newer SGLang requires for Conv3d (vision/multimodal). Set this before launching agg_vision.sh and any multimodal script. Required for 0.5.9; still required at 0.5.11.
• Local model cache: confirm HF_HOME / HF_HUB_CACHE point to a fast disk (/ephemeral/cache on this box) so tests don't redownload weights.
• Verify HF_TOKEN before launch. Anonymous HF requests get 429-rate-limited fast, and gated models (black-forest-labs/FLUX.1-dev, anything with a license click-through) refuse outright. hf auth whoami must succeed; if it errors with "Invalid user token" the env's HF_TOKEN is stale and the user has to provide a fresh one before image_diffusion / multimodal scripts will work.
• Pre-download the heavy / gated models. Letting the launch script trigger the download is fragile under HF rate limits — a half-completed download will error mid-init with a confusing 429 traceback. Pre-fetch with hf download <repo> --token "$HF_TOKEN" first. Big offenders: FLUX.1-dev (~25 GB, gated), LLaDA2.0-mini-preview (~35 GB), Wan2.1-T2V-1.3B-Diffusers.

Step 8: Walk Every Launch Script

Read first before launching anything:

• components/src/dynamo/sglang/CLAUDE.md — SGLang Backwards Compatibility policy and component layout.
• components/src/dynamo/sglang/_compat.py — current shim. The existing fallback comments tell you what N-1 is today; that's the version about to age out.

Path: /ephemeral/dynamo/examples/backends/sglang/launch/

Run them in roughly this order — simpler first, multi-modal/diffusion last:

agg.sh
agg_embed.sh
agg_router.sh
agg_vision.sh
disagg.sh
disagg_router.sh                    # needs >=4 GPUs; SKIP otherwise
disagg_same_gpu.sh                  # ask the user; often skip
diffusion_llada.sh
image_diffusion.sh
text-to-video-diffusion.sh
multimodal_epd.sh
multimodal_disagg.sh                # needs >=3 GPUs; SKIP otherwise

For each script:

1. Note nvidia-smi GPU count vs. script's GPU need. Skip with a recorded reason if short.
2. pkill -9 -f sglang; pkill -9 -f dynamo; pkill -9 -f sgl_diffusion; sleep 3 before launch. Diffusion workers spawn an sgl_diffusion::scheduler child process that survives pkill -f sglang; explicitly grep for it. After diffusion runs, nvidia-smi may still show ~30 GB used by an orphan — kill it by PID.
3. Tee output: bash launch/<script>.sh 2>&1 | tee /tmp/dyn-sgl-bump-<script>.log. The dynamo frontend logs a 404 GET / every ~10s from a health probe that has nothing to do with you; filter with grep -vE "GET.+uri.+/" when reading.
4. Validate with the matching health/inference probe:
   • chat scripts: curl -s -X POST localhost:8000/v1/chat/completions -d '{...}'
   • embed: /v1/embeddings, expect dim count
   • router: confirm a Selected worker: line shows up in the worker log per request (kv-router decision)
   • vision: send a data:image/png;base64,... URL — public image URLs (Wikimedia, raw.githubusercontent.com test fixtures) often 403 / 404. Use /ephemeral/sglang/examples/assets/example_image.png as the canonical inline image.
   • diffusion/video: check the file at the returned file://... URL is non-empty (~150 KB MP4 / ~200 KB PNG for the small test args)
5. On failure, read the full traceback before guessing — guessing from release notes wastes more time than reading the stack. Then jump to the fix patterns below.
6. After PASS, kill cleanly. Don't leave servers behind between scripts. The dynamo.frontend process is reparented to PID 1 after the bash trap fires; pkill -f sglang won't catch it. Always re-check ps -ef | grep -E "sglang|dynamo|sgl_diffusion" before next launch.

Step 9: Fix Patterns (from 0.5.9 bump — expect similar shapes)

When the same kind of breakage recurs across SGLang releases, it usually fits one of these molds. Match the symptom, then read SGLang source to confirm before patching Dynamo.

Before patching anywhere else, ask: "Is this an SGLang import / API surface change?" If yes, the fix belongs in components/src/dynamo/sglang/_compat.py (Pattern E). Component code must not grow scattered try/except blocks for SGLang imports.

Pattern A — Result type changed from dict to dataclass

Symptom: TypeError: 'GenerationResult' object is not subscriptable or AttributeError: 'dict' object has no attribute 'frames'.

Where it hit in 0.5.9:

• components/src/dynamo/sglang/request_handlers/video_generation/video_generation_handler.py
• components/src/dynamo/sglang/request_handlers/image_diffusion/image_diffusion_handler.py

Fix shape: replace result.get("frames", []) / result["frames"] with result.frames, with None/empty-list guards. Apply the same to any sibling fields the new dataclass exposes.

Pattern B — Required field defaulted to None

Symptom: TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' deep in an SGLang scheduler mixin.

Where it hit in 0.5.9: components/src/dynamo/sglang/args.py — max_running_requests was None for DLLM workers because DLLM's mixin assumed normal-scheduler init had already run.

Fix shape: in args.py, after server-args construction, guard the relevant attribute and supply a sane default (8 for DLLM-style workers in 0.5.9). Use getattr(args, ..., None) so SimpleNamespace test stubs don't break.

Pattern C — ModelType / output modality regression

Symptom: Rust frontend looks for a config.json that doesn't exist (diffusers checkpoints don't ship one), or an image worker registers as a chat model.

Where it hit in 0.5.9: components/src/dynamo/sglang/init_diffusion.py. The default --output-modalities=["text"] collapsed ModelType.Images back to Chat|Completions.

Fix shape: in the per-worker init module, override output_modalities to the correct value (["image"] for image, ["video"] for video, etc.) before model registration.

Pattern D — engine is None for non-LLM handlers

Symptom: AttributeError: 'NoneType' object has no attribute 'tokenizer_manager' when launching encode-only / mm-encode workers.

Where it hit in 0.5.9: components/src/dynamo/sglang/request_handlers/handler_base.py — BaseWorkerHandler.__init__ unconditionally touched engine.tokenizer_manager and engine.async_generate, but MultimodalEncodeWorkerHandler constructs with engine=None (it owns an MMEncoder instead).

Fix shape: guard those attribute accesses behind if engine is not None. Don't push the guard further out — the call sites do legitimately pass None.

Pattern E — SGLang API moved / renamed / signature changed (use _compat.py)

This is the most common shape of breakage across SGLang bumps. Always handle these in components/src/dynamo/sglang/_compat.py — never scatter try/except blocks across handler/init files.

Read components/src/dynamo/sglang/CLAUDE.md ("SGLang Backwards Compatibility" section) and the existing _compat.py before adding anything. The policy in force:

1. Support window: N and N-1. Whatever the new target version is, that becomes N. The previous release becomes N-1 and stays supported via a fallback. Anything older gets deleted in this same PR.
2. Only symbols that have actually broken belong in _compat.py. Don't preemptively shim every sglang.* import.
3. Try new path first, except ImportError, fall back. Example layout (this is the shape, not the current code — the actual _compat.py swings in and out of having branches like this as versions cycle):

   try:
       from sglang.srt.new_module import Symbol  # vX.Y+
   except ImportError:
       # Fallback for sglang <vX.Y. Remove when min supported version is vX.Y+
       from sglang.srt.old_module import Symbol  # noqa: F401
   

4. Every fallback branch carries a "remove when" comment naming the version that retires it. No exceptions — without that line, future bumps can't safely prune.
5. Polyfill only what Dynamo actually calls. When SGLang introduces a new class that older releases don't have, the except ImportError branch defines a minimal stand-in that covers exactly the methods/attributes component code touches. Don't reproduce the full upstream surface.
6. Never gate on sglang.__version__. SGLang's version string doesn't always reflect the internal layout. Use try/except ImportError (for moved symbols) or inspect.signature / getattr probing (for changed signatures — see filter_supported_async_generate_kwargs in the current file).
7. Signature drift gets a wrapper, not a try/except at every call site. Read the current _compat.py for the live wrappers — historically these have included mm_encode (MMEncoder._encode signature change), enable_disjoint_streaming_output (ServerArgs field rename), and get_scheduler_info (multi-attribute probing). Wrappers come and go; what stays is the rule that signature drift never leaks into call sites.
8. Defer CUDA-only imports. _compat.py is loaded on test/CI nodes too. Anything that pulls sgl_kernel (e.g. multimodal encoder internals) must be inside the function body, not at module top.
9. Top-of-module SGLang imports must hold for the pre-commit env too. tests/report_pytest_markers.py mocks an explicit allow-list of sglang.srt.* submodules so collection works in the isolated pre-commit venv (no real sglang installed). When you make a previously-conditional from sglang.srt.foo.bar import ... unconditional in _compat.py, add sglang.srt.foo.bar to the _MOCK_MODULES list in that script — otherwise the Report pytest markers hook will fail collecting every sglang test file. Easy to miss because everything works locally where sglang is installed.

Required cleanup as part of every bump PR: prune branches that fall outside the new N / N-1 window. If pruning leaves _compat.py as trivial re-exports, inline the imports at call sites and delete the file. The compat shim is meant to be temporary; carrying dead branches forever defeats the policy.

In the worklog, list each _compat.py change as its own row — mark added, modified, or removed branches and the version each one targets.

General rule

• One fix = one commit with a message like sglang: fix <symptom> for <script>.
• Reference the script that exposed it. This makes the worklog table self-documenting.
• Avoid drive-by cleanups in these commits — the PR is already wide; reviewers will want each fix readable in isolation.
• _compat.py edits are an exception: bundle related shim changes into one commit (sglang: extend _compat for <ver>) so the shim is reviewable as a coherent unit instead of being touched by every other commit.

Step 10: Deprecate Out-of-Window Compat (REQUIRED)

After all launch scripts pass on the new version, before opening the PR, do the deprecation pass. The N / N-1 policy is only meaningful if the bump PR is also the moment we shed N-2 and older.

After bumping to version <new>:

• N = <new> (the new floor + ceiling target)
• N-1 = the version that was current before this bump (still supported)
• Anything older = must be removed in this PR

Procedure:

1. Open components/src/dynamo/sglang/_compat.py. For every fallback branch and polyfill, read its # Fallback for sglang <ver>. Remove when min supported version is <next>+ comment.
2. Decide its fate:

   Branch's "Remove when ≥"	Action
   <= new>	Remove. New version is now the floor; the fallback is dead code.
   > new (still future)	Keep.
   Untagged	Treat as bug from a prior bump. Either tag it correctly now or remove it.

3. After deletions, also remove:
   • Polyfill classes (e.g. the inline NetworkAddress polyfill once 0.5.10 is the floor).
   • Helper wrappers whose multi-version probing is no longer needed (mm_encode, enable_disjoint_streaming_output, get_scheduler_info) — if all probed paths collapse to one, inline the canonical call at the call sites and delete the wrapper.
   • __all__ entries for symbols you removed.
4. If _compat.py collapses to trivial re-exports (no try/except, no polyfills, no signature probing), delete the file and rewrite the call sites to import directly from sglang.srt..... The shim is meant to be temporary.
5. Run the launch-script walk again on the targets you already passed for any handler that now imports differently. Don't trust that "it imported during step 8" means "it still imports after compat pruning" — call sites may have changed.
6. Update components/src/dynamo/sglang/CLAUDE.md's SGLang Backwards Compatibility section if the policy text references a specific version range. Bump the example # Fallback for sglang <ver> snippet to the new N-1.
7. Update the pre-commit mock list. Open tests/report_pytest_markers.py and find the _MOCK_MODULES (sometimes just a top-level list literal of "sglang.srt..." strings). Any sglang.srt.* submodule you newly imported unconditionally — typically by removing a try/except ImportError in _compat.py and pinning to the canonical path — needs to be added there. Without this entry the Report pytest markers hook fails with ModuleNotFoundError: No module named 'sglang.srt.<thing>' when collecting any sglang test, because the pre-commit venv has no real sglang installed. Always run uvx pre-commit run --all-files after the deprecation pass to confirm.

In the worklog, add a section listing every removed branch / polyfill / wrapper and which version it covered. This makes the deprecation reviewable and lets future bumps see the historical pattern.

This step is not optional. If you skip it, the compat file accrues dead branches forever and the policy degrades into "everything is supported." If you genuinely think a fallback should outlive the N-1 window (e.g. a downstream consumer pinned to an old SGLang), surface that to the user explicitly — don't silently keep the branch.

Step 11: Worklog + Memory

When you have ≥1 launch script result, log to ~/memory/dynamo-upgrade-sglang-<ver>/ (create if missing). Use ~/memory/dynamo-upgrade-sglang-0511/ as the current precedent shape — it covers the bundled-feature case (closing a related issue gated on the same version floor) and the deprecation table; older ~/memory/dynamo-upgrade-sglang-059/ is also valid but predates those patterns.

Required artifacts:

• INDEX.md with project frontmatter (status, repo, last-updated)
• bump-to-sglang-<ver>.md with:
  • Branch + PR + GPU env preamble
  • Final results table: # | Script | Status | Notes
  • One numbered fix section per fix, each citing file:line and root cause
  • A _compat.py deprecation table (one row per removed branch / polyfill / wrapper)
  • An "env deps that surfaced" section if you had to install anything beyond [diffusion] (most fresh venvs need nixl, maturin, plus pytest pytest-asyncio pytest-benchmark to run the unit tests — list whatever bit you so the next bumper doesn't relearn it)

Add a row to ~/memory/INDEX.md. Commit and push memory changes with dynamo-upgrade-sglang-<ver>: <short description>.

Step 12: PR

Before pushing: run uvx pre-commit run --all-files. The Report pytest markers hook walks every test file and fails if a previously-conditional sglang.srt.* import is now unconditional but missing from the mock list (see Step 10.7). Catching this locally is much faster than learning about it from CI.

cd /ephemeral/dynamo
uvx pre-commit run --all-files
git push -u origin <branch>
gh pr create --draft --title "sglang: bump to <ver>" --body "<body>"

Body should link the worklog summary table and call out which scripts were SKIPPED (with reason — usually GPU count) so reviewers don't think they were missed.

Container image tags ship in this PR — if upstream has published them. Confirm the tags from Step 1 actually resolve on Docker Hub (docker buildx imagetools inspect lmsysorg/sglang:v<ver>-runtime returns 0) before editing any container file. If they don't exist yet, defer per the Step 1 contingency — don't write a tag that 404s on pull.

When they do exist, update all of:

• container/context.yaml — runtime_image_tag for both the default and CU130 entries
• container/rendered.Dockerfile — ARG RUNTIME_IMAGE_TAG=v<new>-runtime
• container/compliance/README.md — the lmsysorg/sglang:v<...> table rows
• container/templates/sglang_runtime.Dockerfile — any inline version refs (often a comment + the FROM tag). The mooncake-packaging workaround comment is version-specific — check whether the workaround is still needed against the new image; if upstream fixed it, remove the workaround block (don't just bump the version in the comment).

Use the tags the user gave in Step 1 verbatim — don't paste a "natural-looking" guess. The pip version and the image tag don't always agree (.post1, CU variants, RC suffixes).

Out of scope for this PR (separate follow-ups):

• Drive-by cleanups (rename a default model, fix unrelated TODOs in launch scripts, reformat untouched files). Even when the TODO comment ties to "after sglang vX.Y upgrade", let it ride a separate PR.

Bundling exception: if a feature is gated on the new version floor (e.g. a Grafana panel that needs metrics introduced in the bumped version), the user may explicitly request bundling it in. That's fine — it's not a drive-by, it's a feature that could not ship before this PR. Surface the bundling decision in the PR body and the worklog (see ~/memory/dynamo-upgrade-sglang-0511/ for the #8151 precedent).

Anti-patterns

• Don't pip install sglang==<ver> from PyPI — you lose the ability to grep SGLang source while triaging.
• Don't reuse the previous bump's venv. Stale .so files in site-packages cause non-obvious failures.
• Don't uv pip install -e "python" (no extras) on the SGLang checkout. The diffusion launch scripts will break two scripts in. Always python[diffusion].
• Don't skip the uv pip install maturin nixl preflight. Fresh venv has neither, and maturin develop --uv will command not found while dynamo workers will fail at import on nixl.
• Don't keep editing without re-running the failing script. The fix is only real once the script PASSes end-to-end.
• Don't bundle drive-by infra cleanups (default-model TODOs, renames, unrelated reformatting) into the bump PR. They make review harder and obscure which fix corresponds to which breakage. Exceptions: (a) container runtime image tags — these ship in the bump PR (see Step 12); (b) features gated on the new version floor (e.g. dashboard panels that need new metrics) can be bundled if the user explicitly asks, since those couldn't ship before this PR by definition.
• Don't skip a script silently for non-GPU reasons. Either fix it or record why it was skipped.
• Don't add try: from sglang...new_path except ImportError: from sglang...old_path blocks inside handler / init / register modules. Every SGLang import-shape change goes in _compat.py. The component is supposed to import SGLang either directly (when stable) or from _compat (when version-dependent), never via inline try/except.
• Don't gate on sglang.__version__. Use try/except ImportError for moved symbols and inspect.signature / getattr probing for changed signatures.
• Don't add a _compat.py fallback branch without a # Fallback for sglang <ver>. Remove when min supported version is <next>+ comment. Untagged branches accrete forever.
• Don't leave fallback branches for versions older than N-1. Pruning them is part of the bump, not a separate cleanup.
• Don't push without running uvx pre-commit run --all-files first. The Report pytest markers hook walks every test file in the isolated pre-commit venv and trips on any unconditional sglang.srt.* import that isn't in tests/report_pytest_markers.py's mock list. Local pytest passes (sglang is installed) so the failure only shows up in pre-commit / CI.
• Don't accept ruff/black auto-reformatting of unrelated lines in files you touched. Revert the noise lines before committing — the bump PR is already wide; reformatting unrelated assert isinstance(...) blocks just buries the real changes.