testing

name	testing
description	Nim testing conventions, unittest framework, and C++ compatibility patterns
license	MIT
compatibility	opencode
metadata	{"audience":"developers","workflow":"testing"}

What I do

I provide guidance for writing tests in Nim that:

• Work correctly with the std/unittest framework
• Avoid C++ compilation errors with complex FFI types like TorchTensor
• Follow project conventions for test organization

When to use me

Use this skill when:

• Writing new tests for any workspace module
• Debugging C++ compilation errors in test code
• Organizing test fixtures and test data

The unittest framework

Nim's standard library provides a simple testing framework:

import std/unittest

suite "my module tests":
  test "addition works":
    check 1 + 1 == 2

  test "string handling":
    let result = "hello".toUpperAscii()
    check result == "HELLO"

Key procs:

• suite(name, body) - Group related tests
• test(name, body) - Define a single test
• check(expr) - Assert expression is true, prints failed value on failure
• doAssert(expr) - Like check but raises on failure (use for invariants)
• submitTest(result) - Submit test result from a procedure

Critical pattern: Wrap tests in a proc

The problem

When you declare variables at module scope (top-level) in Nim tests, the generated C++ code uses = {} initialization:

TorchTensor expectedTensor = {};  // This fails!
expectedTensor = myFunction(a, b);

The C++ torch::Tensor type (and other FFI types with cppNonPod) does not accept brace initialization. This causes:

error: ambiguous overload for 'operator=' (operand types are 'at::Tensor' and '<brace-enclosed initializer list>')

The solution

Always wrap test code in a proc main():

import std/unittest, workspace/libtorch

proc generateTensor(): TorchTensor =
  # This works - Nim generates:
  # auto result = myFunction(a, b);
  arange(10, kFloat32)

proc runTests*() =
  suite "tensor tests":
    test "generate tensor":
      let tensor = generateTensor()
      check tensor.numel() == 10

when isMainModule:
  runTests()

This generates proper C++:

auto tensor = generateTensor();  // No {} initialization

Test organization

Fixture file pattern

Tests that load files should follow this pattern:

import std/unittest, std/os, workspace/safetensors, workspace/libtorch

const FIXTURES_DIR = currentSourcePath().parentDir() / "fixtures"

proc main() =
  suite "safetensors loading":
    test "load fixture":
      let fixturePath = FIXTURES_DIR / "model.safetensors"
      check fileExists(fixturePath)

      var mf = memfiles.open(fixturePath, mode = fmRead)
      defer: mf.close()

      let (st, offset) = safetensors.load(mf)
      check st.tensors.len > 0

when isMainModule:
  main()

Key points:

• Use currentSourcePath().parentDir() / "fixtures" for fixture paths
• Use memfiles.open with defer: mf.close()
• Return early or use continue for missing fixtures

Test constants

Define test parameters as const at module level:

const Patterns = ["gradient", "alternating", "repeating"]
const Shapes: array[4, seq[int64]] = [
  @[int64 8],
  @[int64 4, 4],
  @[int64 2, 3, 4],
  @[int64 3, 2, 2, 2]
]
const TestedDtypes = [F64, F32, F16, I64, I32, I16, I8, U64, U32, U16, U8]

Helper procedures

Extract reusable logic into proc with * export:

proc generateExpectedTensor*(pattern: string, shape: seq[int64], dtype: ScalarKind): TorchTensor =
  let shapeRef = shape.asTorchView()
  let numel = shape.product()

  case pattern
  of "gradient":
    arange(numel, dtype).reshape(shapeRef).to(dtype)
  of "alternating":
    let flat = arange(numel, kInt64)
    let modVal = (flat % 2).to(kFloat64)
    modVal.reshape(shapeRef).to(dtype)
  else:
    raise newException(ValueError, "Unknown pattern: " & pattern)

Note: Each branch of a case must assign to result.

Running tests

Each module has a task defined in config.nims for running its tests:

# Test toktoktok
nim test_toktoktok

# Test libtorch
nim test_libtorch

# Test safetensors
nim test_safetensors

The command nim test_toktoktok compiles and runs all test files in workspace/toktoktok/tests/ that start with test_ or t_.

Compilation settings

The project uses:

• --path:. - Makes workspace/module imports work
• Tests compiled with: nim cpp -r plus flags for output and cache directories

Fixture files

For this project, fixtures are in:

workspace/toktoktok/tests/tokenizers/

Reference fixtures using:

const FIXTURES_DIR = currentSourcePath().parentDir() / "tokenizers"

Common errors and fixes

"undeclared field" with parameter shadowing

If you have a parameter named shape and access a field info.shape:

proc generateExpectedTensor*(pattern: string, shape: seq[int64], ...): TorchTensor =
  for info in tensors:  # error: 'shape' shadows info.shape
    check info.shape == shape

Fix: Rename parameter to avoid shadowing:

proc generateExpectedTensor*(pattern: string, shapeSeq: seq[int64], ...): TorchTensor =
  for info in tensors:
    check info.shape == shapeSeq  # Now works

Case statement not returning

Each branch of a case must explicitly assign to result:

proc foo(x: int): int =
  case x
  of 1: result = 10  # Must use 'result ='
  of 2: 20             # ERROR: doesn't assign!

How to add a new test

Step 1: Create the test file

Follow the naming convention: test_*.nim or t_*.nim in the module's tests/ directory.

# workspace/my_module/tests/test_myfeature.nim

import std/unittest, std/os
import workspace/my_module

proc runMyFeatureTests*() =
  suite "my feature tests":
    test "basic functionality":
      let result = myModule.function()
      check result == expectedValue

when isMainModule:
  runMyFeatureTests()

Step 2: Run the test

The test will be discovered automatically by the test command:

# If it's in my_module:
nim c -r --task:test_my_module

Or run all tests for the module:

nim test_my_module

Step 3: Add test fixtures (if needed)

Create a fixtures/ directory and add test data:

workspace/my_module/tests/fixtures/

Reference in test code:

const FIXTURES_DIR = currentSourcePath().parentDir() / "fixtures"
let fixturePath = FIXTURES_DIR / "test_data.bin"

Checklist for new tests

• File name starts with test_ or t_
• Located in workspace/module/tests/
• Test code wrapped in proc runTests*()
• Has when isMainModule: runTests() at the end
• Uses defer for resource cleanup (files, etc.)
• Helper procedures exported with *
• Constants defined at module level with const

Python test vector generation

For AI/ML modules, test vectors are generated via Python scripts using torch and safetensors.

Directory structure

workspace/module/
├── tests/
│   ├── test_module.nim          # Nim tests
│   ├── fixtures/                # Generated fixture files
│   │   ├── model.safetensors
│   │   └── tokenizer.json
│   └── testgen/                 # Python test vector generators
│       └── generate_vectors.py

Convention

• Python 3.12 standard for all test vector generation (matches vLLM/SGLang)
• Single root pyproject.toml with [dependency-groups] for shared dependencies:

  [dependency-groups]
  test-vectors = [
      "torch>=2.0.0",
      "safetensors>=0.7.0",
      "transformers>=4.40.0",
      "numpy>=2.4.2",
  ]
  

• Run generators with: uv run --group test-vectors python workspace/module/tests/testgen/generate_vectors.py

Example testgen script

import torch
import numpy as np
from safetensors.numpy import save_file
import os

FIXTURES_DIR = os.path.join(
    os.path.dirname(os.path.dirname(__file__)),
    "fixtures",
)

def generate_vandermonde():
    x = torch.arange(1, 6, dtype=torch.float32)
    vandermonde = torch.vander(x, increasing=True).T
    return vandermonde.to(torch.bfloat16).view(torch.uint16).numpy()

def main():
    fixtures = {
        "BF16_vandermonde_5x5": generate_vandermonde(),
    }
    save_file(fixtures, os.path.join(FIXTURES_DIR, "vandermonde.safetensors"))
    print("Fixtures generated")

if __name__ == "__main__":
    main()

Fixture regeneration

When adding new test vectors, regenerate the fixture files:

uv run --group test-vectors python workspace/module/tests/testgen/generate_vectors.py

Test utilities for libtorch tests

Tests involving TorchTensor and other libtorch FFI types should use the shared test utilities:

import workspace/libtorch_testutils

C++ exception handling

Wrap test code that may throw C++ exceptions:

proc testTensorOps(): bool =
  let a = ones(@[2, 3], kFloat32)
  let b = zeros(@[2, 3], kFloat32)
  let c = a + b
  result = c.isDefined()

when isMainModule:
  runCppTest("tensor operations", testTensorOps)  # Handles exceptions automatically

Or use the template directly:

check catchCppExceptions(testTensorOps())

Tensor assertions

assertDefined - Check tensor is initialized:

let tensor = ones(@[2, 3], kFloat32)
assertDefined(tensor)  # Raises if not defined
assertDefined(tensor, "weight")  # Custom name in error

assertShape - Verify tensor dimensions:

let tensor = randn(@[2, 3, 4])
assertShape(tensor, 2, 3, 4)

assertDtype - Verify tensor dtype:

let tensor = ones(@[2, 3], kFloat32)
assertDtype(tensor, kFloat32)

assertAllClose / assertClose - Compare tensor values:

let actual = computeSomething()
let expected = ones(@[2, 3], kFloat32) * 2.0
assertAllClose(actual, expected)  # Default rtol=2e-2, abstol=2e-2
assertClose(actual, expected, rtol=1e-5, abstol=1e-5)  # Custom tolerance

Debug helpers

printTensor - Print tensor with label:

printTensor(myTensor, "Weight matrix")

printTensorShape - Print shape and dtype:

printTensorShape(myTensor, "Input")
# Output: Input:
#         Shape: [2, 3, 4], Dtype: kFloat32

ptrHex - Convert pointer to hex string for aliasing detection:

let tensor = ones(@[2, 3], kFloat32)
echo "data_ptr = 0x", tensor.data_ptr().ptrHex()
echo "shape.data() = 0x", tensor.shape.data().ptrHex()
# Useful for detecting memory aliasing issues

dataPtrHex / shapePtrHex - Convenience wrappers:

let tensor = ones(@[2, 3], kFloat32)
echo "data_ptr = 0x", tensor.dataPtrHex()
echo "shape_ptr = 0x", tensor.shapePtrHex()
# Equivalent to above but more convenient

printTensorShape(myTensor, "Input")
# Output: Input:
#         Shape: [2, 3, 4], Dtype: kFloat32

traceExec - Debug macro to trace execution:

traceExec:
  let a = ones(@[2, 3])
  let b = zeros(@[2, 3])
  let c = a + b
# Prints each statement before executing

Test file structure

Complete example:

# workspace/my_module/tests/test_feature.nim
import
  std/unittest,
  workspace/libtorch,
  workspace/libtorch_testutils,
  workspace/my_module

proc testBasicFunctionality(): bool =
  let input = ones(@[2, 3], kFloat32)
  let result = myModule.process(input)
  
  assertDefined(result)
  assertShape(result, 2, 3)
  result = true

proc testEdgeCase(): bool =
  let input = zeros(@[1], kFloat32)
  let output = myModule.process(input)
  assertAllClose(output, input)
  result = true

when isMainModule:
  runCppTest("basic functionality", testBasicFunctionality)
  runCppTest("edge case", testEdgeCase)

Key points

• Always import workspace/libtorch_testutils for tests with TorchTensor
• Use runCppTest for formatted output with automatic exception handling (captures the C++ stacktrace; do NOT confuse it with crucible's private proc runTest() scope wrapper — see workspace/crucible/AGENTS.md)
• Use catchCppExceptions when integrating with std/unittest check
• Use assertion helpers (assertDefined, assertShape, etc.) for clear error messages
• Use printTensor and printTensorShape for debugging failures
• Test files should be in workspace/module/tests/ directory
• File names should start with test_ or t_

Kernel tests against a reference

A kernel test that compares against a libtorch reference is three clearly separated parts, so a reader always sees which variable comes from the kernel, which from the reference, and where the comparison happens:

1. A proc that computes with the function under check (runs the kernel, returns the output as a tensor).
2. A proc that computes the reference (libtorch over the same inputs).
3. The check*() proc that builds the shared inputs once, calls both, and asserts.

proc kernelUnderTest(xf: seq[float32], M, N: int): F.Tensor =
  ## Runs the kernel on the fp16-rounded inputs; returns the fp16
  ## output converted to fp32.
  var engine = bkMetal.init()
  engine.ingest(kernelMsl)
  ...  # build fp16 buffers from xf, engine.run, read back
  result = toTensor(outF).reshape(M, N)

proc reference(xf: seq[float32], M, N: int): F.Tensor =
  ## The reference: torch op over the same fp16-rounded inputs.
  let xh = toTensor(xf).reshape(M, N).to(kFloat16)
  result = F.some_op(xh.to(kFloat32))

proc checkFeature(): bool =
  ## Kernel output vs the torch reference on one random batch.
  Torch.manual_seed(0x5EED'u64)
  let xf = scaledRand(M, N, 2.0'f32)
  let actual = kernelUnderTest(xf, M, N)
  let expected = reference(xf, M, N)
  echo &"  worst |Δ| = {worstAbsDiff(actual, expected)} (tolerance 5e-3)"
  assertAllClose(actual, expected, rtol = 0.0'f64, abstol = 5e-3'f64)
  result = true

when isMainModule:
  runCppTest("feature vs the torch reference", checkFeature)

Rules:

• Name the two results actual and expected (or kernel/ref), so the kernel-vs-reference split is visible at the assert.
• Both sides derive from the SAME random inputs, with fp16 rounding applied identically (the kernel buffer rounds once; the reference widens the same fp16 values). Never round the reference from the kernel's fp16 output.
• Complex reference construction (per-seq SDPA, gathers, masks) lives inside the reference proc; the check*() proc stays readable.
• A tiny worstAbsDiff(a, b: F.Tensor): float32 helper prints the worst deviation before the assert; the assert carries the tolerance.
• Test documentation scales with test complexity: an element-wise tile op or a kernel-vs-reference match needs a few lines of setup, the reference call, the tolerance, done.

Forbidden: reimplementation and tautological references

A kernel test's reference must be libtorch math over the same fp16-rounded inputs — never a Nim reimplementation of the kernel's algorithm, and never a reference that shares the kernel's own code path.

Reimplementation tests

Rebuilding the kernel's arithmetic in Nim (matmul loops, norms, rope rotations, ...) instead of calling libtorch is banned:

• The duplication can carry the same misunderstanding twice: the test passes while kernel and reference are both wrong.
• It bloats the test to hundreds of lines. The ceiling for a kernel test is ~80 lines (model: manual_silu_and_mul_fp16.nim); the reference call must be a torch op — F.linear, F.rmsNorm, F.scaled_dot_product_attention, ... from the transformers workspace (workspace/transformers/src/layers/linear.nim pattern).

Data-preparation helpers that mirror a storage format (e.g. a dequant decode table that rebuilds an fp16 weight matrix from its packed bits) are allowed — they are fixture-style reconstruction, not the op's arithmetic — but they must live once in a shared helper, not be copied per test.

Tautological references

A reference that cannot diverge from the kernel proves nothing:

• Reusing the kernel's own helper procs as the reference: a shared decode table used by both sides means a wrong decode passes identically on both.
• Deriving the reference from the kernel's output: rounding, reshaping, or post-processing the kernel result and comparing it to itself.
• Asserting only self-consistency (shape, finiteness, determinism) in place of a value comparison against an independent reference.

The reference must be computed independently of the kernel: same random inputs, fp16 rounding applied identically, torch arithmetic.