ModelTC/LightLLM

test-model-qwen3-8b-gsm8k-scenarios

LightLLM Qwen3-8B GSM8K multi-scenario regression: seven isolated api_server configs (baseline, fp8w8a8 quant, tpsp mix, tpsp with dp2 and dp prefill balance, cpu cache, int8kv on top of cpu cache, disk cache with LIGHTLLM_DISK_CACHE_PROMPT_LIMIT_LENGTH).…

Quellsprache: Chinesisch

test-model-common

Common override guidance for all skills/test_model sub-skills. Applies to LightLLM model accuracy/speed tests that use lm_eval or lmms_eval, especially local-completions GSM8K runs.

Quellsprache: Chinesisch

test-model-qwen3-5-0-8b-pd-nixl

LightLLM Qwen3.5-0.8B PD disaggregation over NIXL gsm8k: pd_master on 8089, prefill on 8001, decode on 8002. Supports TP1 and TP2 runs by setting TP / PREFILL_CUDA_DEVICES / DECODE_CUDA_DEVICES. Qwen3.5 has linear-attention state transfer; use…

Quellsprache: Chinesisch

test-model-qwen3-8b-pd-nixl

LightLLM Qwen3-8b PD disaggregation gsm8k: pd_master on 8089, prefill on 8001, decode on 8002, tp 2 each. Assign four GPUs via nvidia-smi then export PREFILL_CUDA_DEVICES / DECODE_CUDA_DEVICES (no fixed card IDs; no complex shell automation). UCX_NET_DEVICES…

Quellsprache: Chinesisch

lightllm-profiler-control

LightLLM profiler 使用说明。用于需要启动或停止 LightLLM 的 torch_profiler / nvtx profiling 功能时，尤其是查看 --enable_profiling、/profiler_start、/profiler_stop 的使用方法。

Quellsprache: Chinesisch

test-model-deepseekv32-ep

Runs LightLLM DeepSeek-V3.2 EP MoE gsm8k: api_server with --tp 8 --dp 8 --enable_ep_moe, tool_call_parser deepseekv32, reasoning_parser deepseek-v3, graph_max_batch_size 32, mem_fraction 0.8, LOADWORKER 14, port 8000 aligned with lm_eval base_url. Requires a…

Quellsprache: Chinesisch

test-model-deepseekr1-mtp-tp

DeepSeek-R1 MTP-TP test: LightLLM api_server with MTP (EAGLE) draft, tensor parallel only (--tp 8, no --dp, no EP MoE), plus GSM8K lm_eval on localhost. Distinct from the MTP-EP-TPDP skill which uses --tp 8 --dp 8 and EP MoE. Requires a dedicated log…

Quellsprache: Chinesisch

test-model-deepseekr1-base-tp

Runs LightLLM DeepSeek-R1 baseline TP gsm8k: single api_server with --tp 8 and --batch_max_tokens only, no MTP draft, no --dp, no EP MoE (distinct from deepseekr1-mtp-tp which adds MTP). GSM8K lm_eval on localhost port 8089. Requires a dedicated log…

Quellsprache: Chinesisch

test-model-deepseekr1-mtp-ep

Runs LightLLM DeepSeek-R1 EP MoE + MTP (EAGLE) server variants and GSM8K lm_eval against localhost. Requires each full run to use a dedicated log directory: persist every api_server process log under that tree (per-variant subdirectories recommended), write…

Quellsprache: Chinesisch

test-model-qwen2-5-14b-fp8kv-gsm8k

LightLLM Qwen2.5-14B-Instruct GSM8K with FP8 KV cache quantization: either fp8kv_sph (per-head calibration JSON) or fp8kv_spt (per-tensor calibration JSON). Single api_server tp 2 fixed HTTP port 8089 (not configurable), lm_eval local-completions. Assign GPUs…

Quellsprache: Chinesisch

test-model-qwen3-5-0-8b-gsm8k-scenarios

LightLLM Qwen3.5-0.8B GSM8K multi-scenario regression: five isolated runs (baseline api_server, prefill cudagraph, linear-attention cache flags, CPU cache plus linear-att, disk cache with LIGHTLLM_DISK_CACHE_PROMPT_LIMIT_LENGTH). Each scenario uses api_server…

Quellsprache: Chinesisch

test-model-qwen3-vl-8b-mmmu-val

LightLLM Qwen3-VL-8B-Instruct: api_server tp 2 on port 8089, then lmms-eval CLI (python -m lmms_eval, model openai_compatible, tasks mmmu_val, batch_size 900) with OPENAI_API_BASE pointing at LightLLM OpenAI-compatible /v1. Restore https_proxy for Hub while…

Quellsprache: Chinesisch

test-model-qwen3-vl-8b-vit-sep-mode

LightLLM Qwen3-VL-8B-Instruct visual separation (ViT sep / proxy): three processes in order—config_server on 8090; internal Redis on 6000; visual_only with visual_rpyc 8091 and afs_image_embed_dir; normal api_server tp 2 port 8089 with visual_use_proxy_mode.…

Quellsprache: Chinesisch