wally-e2e

# Wally e2e Entry: `scripts/test/e2e.sh <path-to-wally>`. Always runs `scripts/test/smoke.sh`, then `scripts/test/e2e-modalities.sh` (engine-agnostic primitives). Public CI leaves modality knobs unset so every round-trip **skips**. Device runs set `WALLY_E2E_<MOD>` / `WALLY_E2E_MODEL_ROOTS` / `WALLY_E2E_AUTO=1`. See `wally-device-e2e` for ANE/NPU. | Env | Primitive | Example | |---|---|---| | `WALLY_E2E_LLM` / `WALLY_E2E_MODEL` | llm | `mlx-qwen3` or a `*_HNPU` dir | | `WALLY_E2E_STT` | stt | `whisper-tiny` or `whisper_base_HNPU` | | `WALLY_E2E_TTS` | tts | `piper` or `kitten_micro_0_8_HNPU` | | `WALLY_E2E_VLM` | vlm | `smolvlm2` (SDK inserts the media marker) | | `WALLY_E2E_EMBED` | embed | `minilm` or `embeddinggemma_300m_HNPU` | | `WALLY_E2E_IMAGE` | image | compiled SD1.5 tree / `sd15` | | `WALLY_E2E_NEURT_MODEL` | classified by path | `sd15`, a Parakeet ANE tree, or `lfm2-230m-ane` | | `WALLY_E2E_VAD` | vad | `silero` | | `WALLY_E2E_RERANK` | rerank | `bge-reranker` | | `WALLY_E2E_SEGMENT` | segment | `segformer` (P6 PPM) | | `WALLY_E2E_ENGINE` | override only | `qhexrt` / `neurt` / `mlx` | Legacy `WALLY_E2E_MLX_MODEL` / `WALLY_E2E_NEURT_MODEL` / `WALLY_E2E_QHEXRT_MODEL` are classified by path/id into a primitive (not always image). Do not add new engine-named knobs. `scripts/test/assert-binary-backends.sh` greps `nm`/`llvm-nm`/`dumpbin`/`strings` for registrar symbols (`raMLXRegisterRuntime`, `rac_plugin_entry_neurt`, `rac_plugin_entry_qhexrt`, …) so a backends() listing cannot pass without the engine actually being linked into the bottle. Pass `WALLY_SDK_KIT` so overlay backends (`neurt` / `qhexrt`) are required when those libs are in the kit. `CMAKE_PREFIX_PATH` is only used for `HAS_*` flags and Windows DLL staging — an ambient overlay prefix must not make a public OSS bottle fail for missing NeuRT. ## What "green" means `scripts/test/assert-backends.sh` requires every engine the kit actually ships: | Condition | Required `wally --json backends` name | |---|---| | no kit Config (public OSS bottle) | `llamacpp` + `onnx` + `sherpa` | | kit `RunAnywhere_HAS_LLAMACPP` TRUE | `llamacpp` | | kit `RunAnywhere_HAS_ONNX` TRUE | `onnx` | | kit `RunAnywhere_HAS_SHERPA` TRUE | `sherpa` | | Darwin arm64 product binary `wally` (not `wally-cxx`) | `mlx` | | overlay `lib/librac_backend_neurt.a` / `rac_backend_neurt.lib` | `neurt` | | overlay `lib/librac_backend_qhexrt.a` / `rac_backend_qhexrt.lib` | `qhexrt` | Do **not** drop sherpa from the expected list to make 0.20.26 Windows green while `HAS_SHERPA` is TRUE. That kit compiled sherpa with speech ops off (`RAC_SHERPA_ROUTABLE=0`): `rac_backend_sherpa_register()` returned SUCCESS, `capability_check` returned `BACKEND_UNAVAILABLE`, the registry refused the plugin. The fix is pin a routable kit (0.20.28+), not weaken the assertion. ## `backends` must walk every live primitive `src/commands/cmd_backends.rs` iterates `1 .. RAC_PRIMITIVE_COUNT-1`, skipping retired wire value 6. ONNX without RAG only advertises SEGMENT / DIARIZE. A hardcoded GENERATE_TEXT / TRANSCRIBE / EMBED list made onnx invisible even when the plugin was registered. Do not reintroduce a primitive allow-list. ## Windows DLLs Win32 `LoadLibrary` searches the exe directory, then PATH. `e2e.sh` copies `third_party` / `bin` / `lib` `*.dll` next to `wally.exe` and prepends those dirs to PATH **before** smoke. Skipping that produces "llamacpp only" even when the kit contains `rac_backend_onnx.lib` + `onnxruntime.dll`. Never pass `onnxruntime.dll` to `link.exe` (LNK1107) — link the import lib; stage the DLL at runtime. GitHub Windows: `GITHUB_WORKSPACE` is `D:\a\...`; msys `tar -C` needs `cygpath -u` (`fetch-kit.sh` already does). ## Apple MLX host link `cmake/WallyRust.cmake` gets the kit's link line without hand-parsing Ninja: it queries the CMake file API (`cmake_file_api(QUERY API_VERSION 1 CODEMODEL 2)`) against `wally_link_probe`, a target configured but never built that carries the same kit closure the old C++ `wally` executable had. `build.rs` (`link_native`/`probe_link_args`) reads that reply, drops compile-only fragments (`-D`/`-I`/`-O`/…), hands the rest to `cargo` as `rustc-link-arg`s for every artifact it links, and — when `WALLY_NATIVE_LINK_ARGS_OUT` is set (CMake sets it) — writes the same list to `build/wally-native-link-args.txt`. `scripts/build/build-mlx.sh` reads that file and turns each fragment into an `xcodebuild` `OTHER_LDFLAGS` token, then links `build/cargo/release/libwally.a` (the crate's staticlib, built with the same fragments) against it. No manual `ninja -t commands` harvesting, no `bundle-core.sh` merge step — both are gone. - swiftc (Xcode 27) rejects raw `-Wl,` options and ignores bare archive paths in `OTHER_LDFLAGS`, so `build-mlx.sh` sends everything aimed at `ld` through `-Xlinker`; `-l`/`-L`/`-F` and `-framework` are swiftc options and pass through as-is. `scripts/build/build-mlx.sh` must dump the xcodebuild log on failure (`Undefined symbols` does not contain `error:`). Do not grep bare `error:` — every CompileC line contains `-Werror=`. Observed CI `32786359915`: grep `error:|Metal|BUILD` left only `clang: error: linker command failed`. Never put `#` comments in a `\`-continued `xcodebuild` invocation. Bash cuts the command there, so `OTHER_LDFLAGS` and the log redirect never run (empty `xcodebuild-mlx.log`, status taken from a later assignment). Link flags that must survive the Swift host: - The kit's plugin backends (static registrars) arrive already force-loaded in the fragments CMake recorded for `wally_link_probe` — `build-mlx.sh` does not re-derive which archives need `-force_load`, it replays what CMake linked. - `-L$KIT/third_party -lonnxruntime` and `-Wl,-rpath,$KIT/third_party` must both survive the flag rewrite, or the Swift host abort-traps at launch (`Library not loaded: @rpath/libonnxruntime.dylib`). - The crate's own native dependencies (Rust std, native-tls's `Security.framework`, `-liconv`) and the C++ runtime (`-lc++`, the kit links through the C++ driver) go on the link line after the kit's own flags (`build-mlx.sh`'s `rust_native` array). - Canonicalize `WALLY_SDK_SWIFT_PATH` with `cd && pwd`. SwiftPM's local package identity is the **directory name**, so `…/EXTERNAL/Wally/../..` registers as `..`. Nested checkouts named `sdks1` must use that name in `.product(..., package:)`. Do not point `WALLY_SDK_SWIFT_PATH` at an unreleased `Package.swift` whose `sdkVersion` zips 404 (`v0.20.28` before publish). CI checks out the **tagged** SDK tree (`ref: v$SDK`) whose binaryTargets already exist. CI macOS runner is **macos-26** (Xcode 26 / Swift 6.2). The MLX host resolves `RunanywhereAI/runanywhere-sdks` `Package.swift`, which is `swift-tools-version: 6.2`. macos-15 is Xcode 16.4 / Swift 6.1 and fails after a successful libtool merge with `package 'runanywhere-sdks' is using Swift tools version 6.2.0 but the installed version is 6.1.0`. macos-14 is Swift 5.10. `release.yml` must use the same runner as `ci.yml`. ## Linux Linux bottles are not a v1 merge blocker. Windows x64 and macOS arm64 are. `scripts/test/e2e-linux.sh` exists for later. ## Private engines NeuRT / QHexRT only appear in `backends` when the overlay was applied. `scripts/test/e2e.sh` requires `neurt` / `qhexrt` when `lib/librac_backend_neurt.a` or `lib/rac_backend_qhexrt.lib` exists — not by grepping packaged `HAS_NEURT FALSE` (that stays false; find_package flips it when the archive is present). Public CI must pass without overlays. Image gen (`cmd_image.rs`, gated `#[cfg(wally_has_neurt)]`) is compiled out unless NeuRT is present. ## Device / overlay gotchas (0.5.1 + kit 0.20.28) Public bottles never list `neurt` or `qhexrt`. That is the product, not a test gap. Overlay-rebuild the product binary (`WALLY_APPLE_MLX_HOST=ON` on Mac; ARM64 MSVC + QHexRT overlay on Snapdragon). - **`CMAKE_PREFIX_PATH` is not an overlay opt-in.** Only `WALLY_SDK_KIT` makes e2e require `neurt`/`qhexrt`. An ambient overlay prefix from a previous rebuild will otherwise fail a public-bottle run. - **Binary assert:** never `nm | grep -q` under `pipefail` (SIGPIPE → false FAIL). Stream `strings -a` / `nm -a`. Darwin MLX proof is `mlx-swift_Cmlx.bundle` next to product `wally` (`nm -gU` misses Swift host symbols). First C++ `rac_plugin_register(mlx)` logs `-811`; Swift callbacks then register MLX — noisy, not a miss. - **Windows ARM64 public/overlay kits have `HAS_LLAMACPP FALSE`.** Do not require `llamacpp` in e2e. Overlay `wally.exe` listing **only** `qhexrt` (priority 150) is correct. On-disk GGUF (`qwen3.5-2b`) cannot run there. - **QHexRT generate needs QAIRT matching the device skel, not the overlay DLL set.** Snapdragon X2 Elite / Hexagon v81: `QNN_SDK_ROOT` + `ADSP_LIBRARY_PATH=%QNN_SDK_ROOT%\lib\hexagon-v81\unsigned`, copy `aarch64-windows-msvc` `QnnHtp.dll` / `QnnHtpPrepare.dll` / `QnnHtpV81Stub.dll` / `QnnHtpV81CalculatorStub.dll` / `QnnSystem.dll` next to `wally.exe`. Overlay 2.47 DLLs vs device 2.41 skels fail; QAIRT **2.48** worked. Pass the `*_HNPU` directory (`--engine qhexrt`), not a GGUF. FastRPC `openSession` timeouts (~90s) then user-driver fallback are normal; a second generate while DSP is wedged fails with `Skel failed to process context binary` / `0x3ea` — `taskkill wally.exe` and use a `.bat` with **fully expanded** `ADSP_LIBRARY_PATH` (nested `%QNN_SDK_ROOT%` in `cmd /c "set A=…&& set B=%A%\…"` does not expand). - **VS on the ARM64 box may be 2026 / 18 Community**, not 2022: `C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvarsarm64.bat`. CMake/Ninja live under VS CMake extensions; they are not on default PATH. - **v0.20.28 Windows ARM64 public kit omits `libcurl.lib`.** Copy from `arm64-windows-static` into the kit `lib/` before linking (fixed in the SDK packager for the *next* kit; do not retag 0.20.28). Wally already links kit `libcurl.lib` when present. - Published product bottles: macOS `wally-$V-macos-arm64.tar.gz`, Windows **x86_64** zip, Windows **arm64** zip, and Linux **x86_64** `wally-$V-linux-x86_64.tar.gz`. NPU (NeuRT/QHexRT) is overlay-only on any platform. - **The private QHexRT overlay tarball used to ship zero skel files** (only `.dll`/`.lib`, no `.so`/`.cat`) — `rac-cli`'s own overlay build could not run `qwen3.8-27b-1bit-npu` (the Bonsai/Maple ternary decoder) out of the box; validating it required hand-copying `librun_main_on_hexagon_skel.so` + `.cat` in from the `electron-qhexrt` npm package as a workaround. Fixed in `runanywhere-sdks`' `scripts/build/package-private-engine-overlay.sh` (widened the copy filter and added a pass for `dsp/win-arm64/`). **Wally itself never had the `ADSP_LIBRARY_PATH` bug the Electron binding had** — `fastrpc_win.cpp`'s `exe_dir()` fallback naturally resolves for `wally.exe` because dependent DLLs/skels are staged flat beside the executable by this repo's own packaging convention — but that protection is a property of the *packaging layout*, not of Wally's code, so it is not something to assume going forward. **Always build a fresh overlay from the actual release script and run the ternary model against it after any SDK kit-pin bump** that touches QHexRT — do not assume last time's manually-patched overlay is still representative of what a real user's overlay build produces. - **The private overlay tarball must be EXTRACTED ON TOP OF `kit/`, merging into the same directory tree (`overlay/bin/*` → `kit/bin/`, `overlay/lib/*` → `kit/lib/`, `overlay/include/*` → `kit/include/`, `overlay/share/...` → `kit/share/...`) — never kept as a separate sibling `overlay/` directory fed to CMake via a second `CMAKE_PREFIX_PATH` entry.** `wally_stage_windows_runtime_dlls()` (`cmake/RunAnywhereSDK.cmake`) only ever copies from `${RunAnywhere_LIBRARY_DIR}/../bin` — i.e. `kit/bin` — so a same-named `overlay/bin` sitting next to `kit/` is silently never consulted. Worse, this fails **completely silently**: the build succeeds, `wally.exe` links, and `wally backends --json` returns `{"backends":[]}` with no error naming QHexRT at all (`find_library`-style detection in `RunAnywhereSDK.cmake` just doesn't find `kit/lib/rac_backend_qhexrt.lib` because it was never copied there). If a fresh overlay build reports zero backends, check this BEFORE suspecting the overlay tarball's contents. - **`qwen3.8-27b-1bit-npu`'s `HostOpFailed` had THREE compounding causes, found and fixed one at a time — a kit-pin bump to v0.20.31 alone was NOT enough; Wally needed its own additional fix (below) even with a perfectly merged overlay.** 1. The overlay-skel-files-never-shipped bug (above), fixed upstream in `runanywhere-sdks`' overlay packaging script. 2. `qhexrt::qnn::Backend::profile()` — called (via the same `engines/qhexrt/qhexrt_session.cpp` this repo statically links, same as the Electron binding) to pick the `v75`/`v79`/`v81` manifest directory before the manifest is even parsed — shared its device query with the code path that opens a real QNN HTP device, so the ternary decoder's `host_only` manifest paid for a live QNN device it never needed. Fixed in `neurun` v0.20.31 (`Backend::profile()` no longer shares `ensure_device()` with `device()`) — see that repo's `qhexrt-profile-must-not-create-live-device` KB finding.