| name | cherry-pick-eval |
| description | Evaluate upstream PRs or commits individually against the K2.6 baseline. Tests each change in isolation using the same L0-L4 pipeline, identifying which are helpful, harmful, or neutral for our specific workload. Use when new upstream PRs land (Alpha-MoE, Mega MoE, FP8 fusion) or when evaluating a bundle of commits. Never assume a bundle is net-positive — the EFA harness found bundled expert commits regressed LL performance by 10%. |
Cherry-Pick Evaluation
Overview
Evaluates upstream PRs, commits, or patch bundles individually against the K2.6 serving baseline. Inspired by the EFA harness finding: 12 hand-crafted expert commits applied together regressed low-latency performance by 10%, even though 6 of the 12 were individually helpful (−15% to −25% each).
The principle: commit-level granularity matters. A bundle of optimizations can be net-negative if harmful commits outweigh helpful ones for a specific workload profile.
Usage
Input
- PR/commit list: GitHub PR numbers or commit SHAs to evaluate
- Target workload: Which K2.6 workload profile to measure against (W1-W6, or custom)
- Baseline: Current champion configuration (default: vLLM FLASHINFER_MLA)
Procedure
For each PR/commit in the list:
- Isolate: Apply ONLY this single change to the baseline image
- Build: Compile with the change (record build time, any warnings)
- Verify: Run L2 correctness check (catch accuracy regressions early)
- Benchmark: Run L4 e2e benchmark at c=128 and c=512
- Classify:
- Helpful: Speedup CI excludes 1.0 (improvement confirmed)
- Harmful: Slowdown CI excludes 1.0 (regression confirmed)
- Neutral: CI includes 1.0 (no measurable effect)
- Record: Classification + telemetry for each change
Output
{
"evaluation_id": "cherry_pick_001",
"baseline": "vllm_v0.19.1_flashinfer_mla",
"workload": "W3_agentic_multiturn",
"commits_evaluated": [
{
"ref": "vllm#35474",
"title": "Dynamic MLA/MHA Routing for Sub-1024 Token Prefill",
"verdict": "helpful",
"throughput_delta": "+8.3%",
"ttft_delta": "-62%",
"ci": [1.04, 1.13],
"notes": "Significant TTFT improvement on short agentic prompts"
},
{
"ref": "vllm#30078",
"title": "Alpha MoE integration",
"verdict": "helpful",
"throughput_delta": "+11.2%",
"ci": [1.07, 1.16],
"notes": "16% kernel speedup translates to 11% e2e on K2.6 384-expert"
},
{
"ref": "vllm#36297",
"title": "Fused BMM+FP8 quant for MLA v_up_proj",
"verdict": "neutral",
"throughput_delta": "+0.4%",
"ci": [0.98, 1.03],
"notes": "Kernel-level 1.7x but <1% e2e — v_up_proj is not the bottleneck"
}
],
"composition_recommendation": "Apply #35474 + #30078. Skip #36297 (neutral). Test composition for interactions."
}
Composition Testing
After individual evaluation, test pairwise and full compositions of helpful commits:
- Do two helpful changes interact positively (super-linear) or negatively (contention)?
- The EFA finding: helpful changes can conflict when combined
Constraint Database Integration
- Helpful PRs: Extract the optimization idea, inject as positive constraint into
/generate-candidate
- Harmful PRs: Record as soft constraint ("this approach regresses K2.6 because...")
- Neutral PRs: Record with evidence — prevents re-evaluation of the same approach
Priority PRs to Evaluate (Phase 1)
| PR | What | Why Test on K2.6 |
|---|
| vLLM #35474 | Dynamic MLA/MHA routing (<1024 tokens) | ~3x TTFT for agentic workloads |
| vLLM #30078 | Alpha-MoE (16% over DeepGEMM on DS V3) | Unknown on 384-expert layout |
| SGLang #23167 | DeepGEMM Mega MoE integration | Tuned for 256 experts, test on 384 |
| vLLM #35879 | RoPE+KV cache fusion (+16% TPOT) | Tested on H100, not B300 |
| vLLM #36297 | Fused BMM+FP8 for MLA decode | 1.7x kernel-level, unknown e2e |
When NOT to Use
- For agent-generated kernels (use
/verify-kernel instead)
- When evaluating ideas from PRs (inject as constraints, don't cherry-pick code)
- For PRs that require major refactoring to apply (out of scope for A/B testing)
