trtllm-case-executor

This skill executes TensorRT-LLM test commands by checking the environment and dispatching to the appropriate executor. All commands are accepted as pre-built strings — this skill does **not** build or modify commands. **Four workflows** (classified in this order — first match wins): 1. **Custom** — `custom_cmd` or `custom_script` provided. Runs arbitrary commands via `slurm_run_custom.sh`. 2. **Bench** — `bench_cmd` provided. Accepts a pre-built `trtllm-bench` command chain. Parses `--tp`/`--pp` for `required_devices` only; does not modify the command. 3. **Eval** — `eval_cmd` provided. Accepts a pre-built `trtllm-eval` command. Parses `--tp_size`/`--pp_size` for `required_devices` only; does not modify the command. 4. **Pytest / test_cmd** — `test_cmd` provided. Handles all other commands: non-perf pytest, perf-sanity (`test_e2e[...]`), python scripts, and anything else. When `perf_config_yaml` is provided alongside a perf-sanity `test_cmd`, it is passed directly to the script builder for node/GPU sizing; no auto-derivation from the test ID is performed. --- ## Inputs ### Core Inputs | Input | Description | Required | Example | |-------|-------------|----------|---------| | `test_cmd` | Test command to run inside the container (pytest, python, perf_sanity, etc.) | **Yes** (pytest/test_cmd) | `pytest tests/integration/defs/perf/test_perf_sanity.py -v "test_e2e[aggr-gpt_oss-gpt_oss_tp4ep4pp2_mtp1]"` | | `bench_cmd` | Pre-built `trtllm-bench` command string. May chain `prepare-dataset` and `throughput`/`latency` with `&&`. Mutually exclusive with `test_cmd`/`eval_cmd`/`custom_cmd`. | **Yes** (bench) | `trtllm-bench ... prepare-dataset ... && trtllm-bench ... throughput ...` | | `eval_cmd` | Pre-built `trtllm-eval` command string. Mutually exclusive with `test_cmd`/`bench_cmd`/`custom_cmd`. | **Yes** (eval) | `trtllm-eval --model /models/llama --tp_size 4 gsm8k` | | `custom_cmd` | Arbitrary command to run inside the container. Mutually exclusive with `test_cmd`. | **Yes** (custom) | `python3 examples/run_inference.py --model llama` | | `custom_script` | Host script path — takes precedence over `custom_cmd` (becomes `bash <script>`). Auto-mounts parent dir. | No | `/home/user/my_benchmark.sh` | | `custom_env` | Space-separated `KEY=VALUE` pairs for custom workflow only | No | `BATCH_SIZE=32 NUM_WORKERS=4` | | `perf_config_yaml` | Explicit path to a perf-sanity config YAML. When provided, passed directly to the script builder for node/GPU sizing — no auto-derivation from the test ID. | No (perf_sanity) | `/work/gpt_oss_120b_fp4.yaml` | | `model_name` | Short name for container/job naming. Auto-derived from command if not provided. | No | `llama_7b` | | `required_devices` | GPU count. Auto-parsed from `--tp`/`--pp` (bench) or `--tp_size`/`--pp_size` (eval) if not provided. Default: `1`. | No | `8` | | `total_required_devices` | Total GPU processes across all nodes. If not provided, derived per the rules below. | No | `8` | | `required_devices_per_node` | GPU processes per node. If not provided, derived per the rules below. | No | `4` | | `skip_install` | Skip `pip install -e .` when image already has TRT-LLM. Default: `false`. | No | `true` | | `build_project` | Build TRT-LLM from source before running the test. Default: `true`. Set to `false` to skip the build step when the image already contains a current build. | No | `false` | | `container_image` | Override container image URI. If unset, case-executor resolves from `<REPO_ROOT>/jenkins/current_image_tags.properties` using the CPU arch derived from `device_type` (or local env-check). The resolved value is forwarded to every downstream consumer in `job_spec.json` — none of them re-resolve. For an example URI shape, delegate to `internal-env-info` for the `container_registry_example` key when that skill is installed; otherwise ask the user for a sample. | No | (delegated) | | `monitor_timeout` | Wall-clock limit. Accepts `1h`, `30m`, `3600s`, or `HH:MM:SS`. Default: `1h`. | No | `2h` | ### Environment & Slurm Inputs | Input | Description | Default | |-------|-------------|---------| | `node_count` | Slurm nodes. Computed by this skill as `ceil(total_required_devices / required_devices_per_node)` and forwarded in `job_spec.json` to every downstream consumer (`trtllm-test-script-builder`, `exec-remote-slurm`). Caller may override only when an explicit value is required. | computed | | `partition` | Slurm partition | *(required for Slurm)* | | `account` | Slurm account | resolution chain (applied in [Step 2.5](#step-25-detect-slurm-account-and-valid-partitions-slurm-scenarios-only)): caller input → `default_account` from `detect_slurm_env.sh` JSON → delegate to `internal-env-info` for the `default_account` key when that skill is installed → otherwise ask the user. No NVIDIA-internal default is hard-coded in this skill. | | `time_limit` | Slurm walltime | `02:00:00` | | `device_type` | Required GPU type (e.g., `B200`, `GB200`). Auto-routes to remote cluster if local doesn't match. | — | | `checkpoint_path` | Host model checkpoint path | — | | `llm_models_root` | Models root inside container | caller input → delegate to `internal-env-info` for the `default_llm_models_root` key when that skill is installed → otherwise ask the user. No NVIDIA-internal default is hard-coded in this skill. | ### Remote Slurm Inputs | Input | Description | Default | |-------|-------------|---------| | `slurm_cluster` | Remote cluster name — skips local env-check, looks up per-cluster info (`mfa_style`, `default_models_repo`, `default_user_root_dir`, `gpus_per_node`) via `internal-env-info` skill. Connection fields (`ssh_host`, `partition`, `account`, `mounts`, etc.) come from caller-supplied inputs in `job_spec.json` (or `internal-env-info` defaults / ask-the-user fallbacks). When `internal-env-info` is not installed, the executor uses whatever explicit cluster fields the caller passed in `job_spec.json`. | — | | `ssh_host` | Full SSH destination override. When unset, `exec-remote-slurm` Step 2.0 delegates to `internal-env-info` for the `ssh_host_pattern` key to construct the destination from `<slurm_user>` and `cluster_name`; it delegates to the same skill for `ssh_host_example` when documentation needs a sample. When `internal-env-info` is not installed and the user did not provide `ssh_host`, **ask the user** to supply `ssh_host` directly — this skill does not hard-code an NVIDIA-internal login-host pattern. `mfa_style` itself comes from `internal-env-info` Step 1b and is used by `exec-remote-slurm` Step 2 only to pick the SSH path (direct vs MFA), not to build the username; when that skill is absent, `mfa_style` defaults to `null` and `exec-remote-slurm` uses its probe-direct-first-then-MFA fall-back path. | constructed (delegated) | | `user_name` | Raw login username forwarded to `internal-env-info` so it can build the SSH-ready `slurm_user` (applying whatever suffix that cluster's `mfa_style` requires). This skill never builds `slurm_user` itself — it only forwards `user_name` and reads the result back. | `$(whoami)` | | `slurm_user` | SSH/Slurm username override. When set, used verbatim and the cluster-info value is discarded. When unset, this skill passes `user_name` to `internal-env-info` and copies the returned per-cluster `slurm_user` into `slurm_env` as-is. | (from cluster-info) | | `slurm_password` | SSH password (**prompted if not provided** — never stored/logged) | *(prompted)* | | `remote_cwd` | Remote working directory | `<REPO_ROOT>` | ### Device Count Auto-Parsing If the caller passes `total_required_devices` and `required_devices_per_node` directly, use those values verbatim — skip all derivation below. Otherwise derive all three fields (`required_devices`, `total_required_devices`, `required_devices_per_node`) using the rules in the table. Derivation order: `required_devices` first, then the two per-node/total fields. | Workflow | `required_devices` | `total_required_devices` | `required_devices_per_node` | |----------|--------------------|--------------------------|----------------------------| | `bench_cmd` | `tp * pp` parsed from last `trtllm-bench` invocation | same as `required_devices` | same as `required_devices` (single-node only) | | `eval_cmd` | `tp_size * pp_size` parsed from `eval_cmd` | same as `required_devices` | same as `required_devices` (single-node only) | | `test_cmd` (non-perf pytest) | From `skip_less_device(N)` marker in test file; otherwise `1` | same as `required_devices` | same as `required_devices` (always single-node) | | `test_cmd` (perf-sanity) with `perf_config_yaml` | `tensor_parallel_size * pipeline_parallel_size` for the matching `server_configs` entry | same as `required_devices` | `hardware.gpus_per_node` from the YAML | | `test_cmd` (perf-sanity) without `perf_config_yaml` | default `1` | `1` | `1` | | `custom_cmd` | `required_devices` input; default `1` | same as `required_devices` | same as `required_devices` | **Perf-sanity YAML extraction procedure** (applies when `perf_config_yaml` is provided): 1. Read the YAML file and extract `hardware.gpus_per_node` → `required_devices_per_node`. 2. Identify the target server config name from `test_cmd` — it is the bracketed test ID suffix, e.g., `aggr-gpt_oss-gpt_oss_tp4ep4pp2_mtp1` in `test_e2e[aggr-gpt_oss-gpt_oss_tp4ep4pp2_mtp1]`. Match the trailing component against `server_configs[].name` in the YAML. 3. From the matched entry, read `tensor_parallel_size` (default `1`) and `pipeline_parallel_size` (default `1`). 4. `required_devices = tensor_parallel_size * pipeline_parallel_size`. 5. `total_required_devices = required_devices` (total GPU processes = world size). 6. If no matching entry is found, fall back to `required_devices = 1`, `total_required_devices = 1`; keep `required_devices_per_node` from step 1. ### Models Mount Resolution Mount to `<llm_models_root>` (the resolved input — see the `llm_models_root` row above). Host-side source priority: `checkpoint_path` parent → `env_check.default_models_repo` → delegate to `internal-env-info` for the `default_models_repo_host` key when that skill is installed → otherwise stop and ask the user. No NVIDIA-specific path is hard-coded in this skill. --- ## Procedure ### Step 1: Create Work Directory and Classify ```bash WORK_DIR="<REPO_ROOT>/work_dirs/<model_name>_$(date +%Y%m%d_%H%M%S)" mkdir -p "$WORK_DIR" ``` Classify workflow (custom → bench → eval → test_cmd). Derive `model_name`, `required_devices`, `total_required_devices`, and `required_devices_per_node` per the [Device Count Auto-Parsing](#device-count-auto-parsing) rules above. If the caller already passed `total_required_devices` and `required_devices_per_node`, skip derivation and use those values directly. Then compute `node_count = ceil(total_required_devices / required_devices_per_node)` (use the caller's value only when explicitly provided). This skill is the **single source of truth for `node_count`** — every downstream consumer reads it from `job_spec.json` and must not recompute. Floor-clamp to `1` when both totals are `1`. Build the resolved `job_name` here too: - `detail = workflow_type` (`pytest`, `bench`, `eval`, `custom`) — or a more specific token derived from `model_name` when available. - `job_name = f"{account}.{detail}"`. This skill is the **single source of truth for `job_name`** — `exec-remote-slurm` (and any other downstream consumer) takes the resolved string from `job_spec.json` and substitutes it directly into `-J <job_name>`. The convention `<account>.<detail>` is documented and applied here only. **Remote work_dir derivation (remote_slurm scenarios only):** Compute the remote-side equivalents of `WORK_DIR` so the remote layout mirrors the local layout (`<REPO_ROOT>/work_dirs/<basename>`): ```text repo_dir_name = basename(repo_url stripped of .git) # e.g., "TensorRT-LLM" remote_repo_path = <default_user_root_dir>/<repo_dir_name> remote_work_dir = <remote_repo_path>/work_dirs/<basename(WORK_DIR)> ```