testing

# Testing Conventions ## Coverage and Ray Actors For any source file under `nemo_rl/*.py` that defines a class or function decorated with `@ray.remote`, add a coverage pragma because these run in separate Ray processes and are not reliably tracked by coverage. Place `# pragma: no cover` on the `class` or `def` line: ```python import ray @ray.remote # pragma: no cover class RolloutActor: def run(self) -> None: ... @ray.remote # pragma: no cover def remote_eval(batch): ... ``` ## Nightly Tests for New Model Support When adding support for a new model, add a corresponding nightly test consisting of: ### 1. Recipe YAML under `examples/configs/recipes/` Place in the appropriate domain subdirectory (`examples/configs/recipes/llm/` or `examples/configs/recipes/vlm/`). Name it following the recipe naming rules below. ### 2. Driver script under `tests/test_suites/` Create a shell script in the matching domain (`tests/test_suites/llm/` or `tests/test_suites/vlm/`). Source any common environment (e.g., `common.env`) and invoke the training entrypoint with `uv run ... --config <path-to-yaml>`. Match the driver script filename to the YAML base name with `.sh`. ### 3. Add to nightly list Append the driver script path (relative to `tests/test_suites/`) to @tests/test_suites/nightly.txt. ## Recipe Naming Rules ### LLM Pattern ``` <algo>-<model>-<nodes>n<gpus>g-<strategy-and-params>[-modifiers][-long][.vN].(yaml|sh) ``` - **algo**: `sft`, `dpo`, `grpo`, etc. - **model**: `llama3.1-8b-instruct`, `qwen2.5-7b-instruct`, etc. - **nodes/gpus**: `1n8g`, `4n8g`, `8n8g` - **strategy-and-params**: `automodel`, `automodel-ep8`, `megatron`, `megatron-ep16tp2cp2` - **modifiers** (optional): `sp`, `actckpt`, `fp8`, `noncolocated`, `quick` - **-long** (optional): long-running recipe - **.vN** (optional): version suffix for convergence-impacting changes Examples: ``` sft-llama3.1-8b-1n8g-automodel-long.yaml grpo-llama3.1-8b-instruct-1n8g-megatron-fp8.yaml grpo-qwen2.5-7b-instruct-4n8g-automodel-tp4sp.v3.yaml ``` ### VLM Pattern ``` vlm_<algo>-<model>-<nodes>n<gpus>g-<strategy>[-modifiers][.vN].(yaml|sh) ``` ### Directory Placement ``` examples/configs/recipes/ llm/<name>.yaml vlm/<name>.yaml tests/test_suites/ llm/<name>.sh vlm/<name>.sh nightly.txt ``` ## Research projects Research unit and functional tests are discovered under `research/*/tests/`. CI syncs each project's runtime dependencies and `test` group before executing them. Research suite scripts belong under `research/<project>/tests/test_suites/`, with matching recipes under `research/<project>/configs/recipes/`. List scripts by their repository-relative paths in the root suite lists (`nightly.txt`, `disabled.txt`, etc.). Script/recipe accounting includes both core and research directories, preserving each research project's identity.