testing

# Testing Skill ## Prerequisites Ensure dev dependencies are installed: ```bash make install # or: uv sync --group dev make develop # build native Rust extension: uv run maturin develop ``` ## Python Tests ### Run the full test suite with coverage ```bash uv run pytest test/ --cov --cov-report=term-missing ``` Or use the Makefile shortcut: ```bash make test ``` ### Run a single test file ```bash uv run pytest test/test_example.py -v ``` ### Run a single test function ```bash uv run pytest test/test_example.py::test_my_function -v ``` ### Useful pytest flags | Flag | Purpose | |---|---| | `-v` | Verbose output | | `-x` | Stop on first failure | | `-k "expression"` | Run tests matching expression | | `-m "not slow"` | Skip tests marked as slow | | `--tb=long` | Full tracebacks | | `--pdb` | Drop into debugger on failure | | `-n auto` | Parallel execution (requires pytest-xdist) | ### Coverage reports ```bash # Terminal report with missing lines (default in make test) uv run pytest test/ --cov --cov-report=term-missing # HTML report (opens in browser) uv run pytest test/ --cov --cov-report=html # Output: htmlcov/index.html # XML report (for CI integration) uv run pytest test/ --cov --cov-report=xml # Output: coverage.xml # Combined reports uv run pytest test/ --cov --cov-report=term-missing --cov-report=html --cov-report=xml ``` Coverage configuration is in `pyproject.toml` under `[tool.coverage.run]` and `[tool.coverage.report]`. The minimum coverage threshold is set to 80%. ### Python benchmarks ```bash # Run benchmarks only uv run pytest test/ --benchmark-only # Run benchmarks with comparison uv run pytest test/ --benchmark-compare # Skip benchmarks during normal test runs uv run pytest test/ --benchmark-skip ``` Benchmark tests should be marked with `@pytest.mark.benchmark` and use the `benchmark` fixture from pytest-benchmark. ## Rust Tests ### Run Rust tests with nextest ```bash cargo nextest run --manifest-path src/rust/Cargo.toml ``` Or use the Makefile shortcut: ```bash make test-rust ``` ### Run specific Rust tests ```bash cargo nextest run --manifest-path src/rust/Cargo.toml -E "test(parse)" ``` ### Rust benchmarks (criterion) ```bash cargo bench --manifest-path src/rust/Cargo.toml ``` Benchmark results are saved to `src/rust/target/criterion/`. HTML reports are generated when the `html_reports` feature is enabled. ## Run everything ```bash make bench # Python benchmarks + Rust benchmarks make check # format-check + lint-check + typecheck + interrogate + deny ``` ## ASV Historical Benchmarks ASV (airspeed velocity) tracks performance across the project's git history, producing an interactive web dashboard. ### Configuration ASV is configured in `asv.conf.json` at the project root. Benchmarks live in `benchmarks/` (separate from the pytest test suite in `test/`). All ASV artifacts (environments, results, HTML) are stored under `.asv/` and gitignored. ### Running ASV benchmarks ```bash # Benchmark the current commit make asv-run # Equivalent to: uv run asv run --quick HEAD^! # Dry-run (smoke test, no results saved) make asv-quick # Benchmark a range of commits uv run asv run v0.1.0..HEAD # Compare two revisions make asv-compare REF1=main REF2=HEAD # Equivalent to: uv run asv compare main HEAD # Find the commit that introduced a regression uv run asv find v0.1.0..HEAD TimePipelineIteration.time_iterate_all # Show results for a specific commit uv run asv show HEAD ``` ### Publishing the dashboard ```bash # Build static HTML dashboard make asv-publish # Output: .asv/html/ # Preview locally make asv-preview # Push to GitHub Pages uv run asv gh-pages ``` ### Writing ASV benchmarks ASV benchmarks use magic name prefixes: | Prefix | Measures | |---|---| | `time_` | Wall-clock execution time | | `mem_` | Memory footprint of returned object | | `peakmem_` | Peak resident memory | | `track_` | Arbitrary numeric value | | `timeraw_` | Execution time in a fresh subprocess | Benchmarks support `setup()`, `teardown()`, `setup_cache()` lifecycle methods and parameterization via `params` / `param_names` class attributes. See `benchmarks/bench_pipeline.py` for working examples.