Run MNN tests / benchmarks on host or real devices. Covers two parallel tracks — (1) the regression / CI suite (static checks, host-side tests, on-device Android arm64 matrix via ./test.sh + test_stages.json) and (2) one-command iOS real-device LLM benchmarking (prefill/decode tok/s, branch comparison). Use when the user asks to run the tests, run CI, smoke-test a build, verify a change on a device, benchmark on-device (Android or iPhone/iPad), or add / select / retune a test stage.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Run MNN tests / benchmarks on host or real devices. Covers two parallel tracks — (1) the regression / CI suite (static checks, host-side tests, on-device Android arm64 matrix via ./test.sh + test_stages.json) and (2) one-command iOS real-device LLM benchmarking (prefill/decode tok/s, branch comparison). Use when the user asks to run the tests, run CI, smoke-test a build, verify a change on a device, benchmark on-device (Android or iPhone/iPad), or add / select / retune a test stage.
MNN Test / CI SKILL (index)
This skill is an index. Pick the document that matches the task and follow it:
Run the regression / CI suite — static checks, host (local) tests, the on-device Android arm64 matrix (./test.sh + test_stages.json); add / select / retune a test stage; audit stale CI scripts; add a new operator test.
Benchmark LLM prefill/decode speed on a real iPhone/iPad (ios_llm_bench.sh); compare branches on iOS Metal/CPU; verify Metal kernel changes on device.
The two tracks are independent: Android/host regression testing goes through
test.sh, while iOS LLM benchmarking goes through
transformers/llm/engine/ios/ios_llm_bench.sh.
Declarative stage matrix. Edit this to add / drop / retune stages — no shell edits needed for the common cases. It is self-documenting via its _documentation block.
# Static checks only:
./test.sh static
# Host regression (CPU only): build + unit suite + smoke + LLM smoke.
./test.sh local# Full on-device matrix on the attached arm64 device:
./test.sh android <serial> # e.g. ./test.sh android R5CY71BJJ9D
<serial> comes from adb devices (the script prefers adbk and falls back
to adb). If the device shows as unauthorized, the user must tap Allow USB
debugging on the phone first.
Running a subset (filters)
Android mode takes an optional filter as the third argument:
./test.sh android <serial> cpu # CPU unit + lowmem + llm
./test.sh android <serial> opencl # OpenCL unit (image+buffer) + opencl smoke
./test.sh android <serial> vulkan # Vulkan unit + vulkan smoke
./test.sh android <serial> gpu # opencl + vulkan
./test.sh android <serial> unit # all unit/op stages only
./test.sh android <serial> lowmem # only the low-memory matrix
./test.sh android <serial> android-ci # bench + smoke + llm only (no unit/lowmem)
Valid filters: all (default) · cpu · opencl · opencl-image ·
opencl-buffer · vulkan · gpu · unit · lowmem · android-ci.
Reading the result (agent-friendly)
Each stage prints a delimited ═══ stage: <name> ═══ block, then a
PASS / FAIL / SKIP line.
A final summary prints total / passed / failed / skipped and one line
per stage. SKIP is not a failure — it means the prerequisite was absent
(e.g. a GPU library, a model, or a missing build artefact).
Exit code is non-zero iff any stage failed. Gate automation on the exit
code, not on log scraping.
Combined stdout/stderr for every stage is saved under
logs/test-<UTC-timestamp>/<stage>.log — read the named log of a failing
stage for the trailing output. rc=137 ≈ OOM-kill, rc=139 ≈ SIGSEGV.
Dynamic-shape device smoke tests
A zero exit code only proves that a backend context ran; it does not prove that
the requested input shape selected the intended dynamic context. For a
shape-sensitive device test, record and validate the runtime-observed input
shape (for example, from a backend dump manifest or the runner's input tensor)
before treating the test as dynamic-shape coverage.
Environment variables
Var
Mode
Meaning
ANDROID_NDK
android
NDK root. Falls back to $HOME/android-ndk-r21.
ANDROID_EXTRA_CMAKE
android
Extra cmake flags appended to the build (e.g. -DMNN_SME2=OFF) — handy for bisecting a backend regression.
LLM_MODEL_DIR
both
Path to an existing on-disk MNN-format LLM model. When set, that directory is used as-is and nothing is downloaded. Defaults to models/<repo-basename>/.
LLM_MODEL_REPO
both
Model repo id for the LLM smoke test. Default taobao-mnn/Qwen2.5-0.5B-Instruct-MNN.
LLM_MODEL_SOURCE
both
Download source when LLM_MODEL_DIR is unset: huggingface (default) or modelscope.
LLM_MODEL_URL_BASE
both
Override the resolve URL prefix outright (wins over LLM_MODEL_SOURCE).
MNN_TEST_SKIP
both
Comma list of exact test names to skip (also set per-stage via the JSON skip field).
Offline / no-network and mainland-China notes
LLM model provisioning is lazy: the download (or LLM_MODEL_DIR check) is
deferred until the llm stage actually runs, and a provisioning failure skips
only that stage. So the unit / smoke / bench stages run fine with no
network.
# Already have the model on disk → no download attempt at all:
LLM_MODEL_DIR=/path/to/Qwen2.5-0.5B-Instruct-MNN ./test.sh local# huggingface.co unreachable (e.g. mainland China) → fetch from ModelScope:
LLM_MODEL_SOURCE=modelscope ./test.sh android <serial>
For the built-in default model the ModelScope org is remapped automatically
(taobao-mnn/* → MNN/*); an explicitly-set LLM_MODEL_REPO is used verbatim.
LLM backend/layout smoke
For backend or tensor-layout optimizations, do not stop at operator tests. Run
an end-to-end llm_demo correctness smoke with a short prompt and another
prompt long enough to cross backend prefill branch thresholds. This catches
real exported-graph layout bugs where an op test covers only the output format,
but the graph also changes an input tensor format.
macOS Metal answer A/B checks
For answer-level comparisons between two macOS Metal runtimes:
A Metal config does not prove that Metal ran. In a restricted execution
context, MTLCreateSystemDefaultDevice() may return nil, after which
llm_demo prints Init Metal Error / Can't Find type=1 backend, use 0 instead and silently produces a plausible CPU answer. Run in a GPU-visible
context and fail the case on either fallback marker.
Use the same exported model, prompt bytes, backend settings and greedy
sampler on both sides. Isolate each runtime's tmp_path / working directory
so cache reuse cannot cross the A/B boundary.
Decide the multi-line prompt contract before running. llm_demo normally
treats every non-empty line as a separate prompt; if one file is intended to
be one long prompt, flatten or use an explicitly one-line-aware runner and
record the transformed-input checksum.
Fixed token limits can stop in the middle of a UTF-8 token and can leave code
answers incomplete. Exact matches are strong evidence, but low character
similarity after an early greedy fork needs semantic review. For code
prompts, use a large enough budget and compile/test the extracted code before
claiming functional correctness.
Configuring stages
Editing test_stages.json is the supported way to
add, drop, or retune unit / lowmem / smoke / bench stages. Every parameter
(forward type, precision, gpuMode, thread count, tag, memory mode,
dynamic-quant option, KleidiAI flag, per-stage skip list, smoke model list,
benchmark args) lives there, and the _documentation block at the top of the
file explains every field and every skip entry's rationale.
Add a stage that runs an existing test in a new config → add an object to
android.stages (or local.stages). See docs/testing.md § "增加专门阶段".
Skip a known-broken test on one stage → add its exact name to that
stage's skip array and document why under _documentation.skip_rationale.
Add a smoke model / bench entry → see docs/testing.md § "新增 smoke 模型或 bench 阶段".
Auditing stale CI/test scripts
When asked to clean up old CI or test scripts, build a usage map before
recommending deletion:
Prefer git ls-files plus targeted rg/git grep over broad filesystem
scans, so generated build directories and local experiments do not look like
maintained CI surface.
Classify scripts by role: active CI entrypoints, declarative test driver,
release/package scripts, manual benchmark helpers, third-party vendored
tests, and local device/debug helpers.
Treat lack of in-repo references as a "review/deprecate" signal, not proof
of dead code; internal CI systems can invoke tracked files by convention.
Prefer a staged deprecation plan unless a script is both unreferenced and
clearly superseded by test.sh / test_stages.json.
When renaming or consolidating test entrypoints, grep for both executable
names and generated-artifact prefixes. Update CI config, .gitignore,
test_stages.json self-documentation, developer docs, skill docs, and code
comments in the same change so the old entrypoint disappears completely.
Adding a new operator test
Write the C++ test under test/op/ (one file, registered with
MNNTestSuiteRegister). The full template + conventions are in
docs/testing.md § "新增算子测试".
If its name prefix matches an existing stage (e.g. op/*), it is picked up
automatically — no JSON change needed. Otherwise add a dedicated stage.
For deeper work on operators themselves, see the
add-new-op skill.
Read next
docs/testing.md is the authoritative deep reference — read it for the per-stage
breakdown, the stage-object field table, and worked examples.
