model-checker

# Model Checker Validates that a HuggingFace model exported via optimum-intel works correctly with OpenVINO GenAI pipelines and passes accuracy benchmarks. ## When to Use - A new model was added to optimum-intel and needs GenAI validation - Verify a HuggingFace model exports to OpenVINO IR and runs inference - Check model accuracy after conversion using who-what-benchmark ## Inputs The user must provide: - **model_id**: HuggingFace model identifier (e.g. `tencent/HY-MT1.5-1.8B`) **or** path to an existing OpenVINO IR directory (e.g. `/tmp/my_model_ir`). When a local path is provided, export is skipped automatically. - **task**: optimum-cli export task. Supported values: - `text-generation-with-past` - `image-text-to-text` - `text-to-image` - `image-to-image` - `feature-extraction` - `text-classification` - `text-to-video` - `automatic-speech-recognition` ## Prerequisites Ensure the Python virtual environment is activated before running any commands. 1. **Locate the virtual environment** — check for common directories at the repository root: `.venv/`, `venv/`, `env/`. Use `list_dir` to find it. If none is found, ask the user for its location. 2. **Check if already activated**: if `which python` or `where python` points inside the virtual environment, it's already activated. If not, proceed to activate it. 3. **Activate** based on the current platform: - **Linux/macOS**: `source <venv_path>/bin/activate` - **Windows (cmd)**: `<venv_path>\Scripts\activate.bat` - **Windows (PowerShell)**: `<venv_path>\Scripts\Activate.ps1` 4. The background terminal doesn't inherit the venv activation. Run it with the venv activated in the same command. ## Procedure ### Step 1: Run check_model.py Run the checker script from the repository root: ``` python3 .github/skills/model-checker/scripts/check_model.py \ --model-id <model_id_or_path> \ --task <export_task> \ --work-dir .model_enabler/<model_type>/model_checker ``` Determine `<model_type>` from the model's `config.json`. When `--model-id` is a path to an existing directory, the script treats it as a pre-converted OpenVINO IR model and automatically skips the export step. Keep each model type in its own work directory so parallel or resumed enablement runs do not overwrite one another's artifacts. Run `python3 .github/skills/model-checker/scripts/check_model.py --help` for the full argument reference including defaults. The `--work-dir` is where all intermediate files, logs, and outputs will be stored. Do not pipe with any additional logging or redirection — the script handles its own logging. #### Skip flags (for re-runs after a fix) When a previous run already passed some steps (e.g. export succeeded but inference test failed), use skip flags to avoid repeating expensive passed steps: - `--skip-export` — reuse existing IR in `<work-dir>/model_ir` instead of re-exporting (avoids re-downloading weights). When `--model-id` is a local path, export is bypassed automatically and that directory is used directly instead of `<work-dir>/model_ir`. - `--skip-llm-bench` — skip the llm_bench inference test - `--skip-wwb` — skip the who-what-benchmark accuracy check - `--skip-wwb-ground-truth` — skip WWB ground-truth collection and reuse `<work-dir>/wwb/gt.csv`; useful when iterating on GenAI target evaluation after ground truth was already collected Do **not** use skip flags on the first run. Only use them when retrying after a targeted fix. ### Step 2: Interpret Results The script logs progress for each step and exits with code 0 (pass) or non-zero (fail). **Pass criteria:** - Export: exit code 0 - Inference test (llm_bench): exit code 0, metrics line logged - WWB accuracy (depends on `--wwb-base` mode): - `--wwb-base optimum` (default): 1. Optimum ground truth generation: exit code 0 2. GenAI target evaluation: similarity ≥ `SIMILARITY_THRESHOLD` - `--wwb-base hf`: 1. HF ground truth generation: exit code 0 2. Optimum target evaluation: similarity ≥ `SIMILARITY_THRESHOLD` 3. GenAI target evaluation: similarity ≥ `SIMILARITY_THRESHOLD` Note: the WWB step is skipped automatically for `automatic-speech-recognition` (no WWB support). **Log files:** each tool writes its own dedicated log; paths are printed during execution. When a step fails, read the corresponding log for the full traceback and context before drawing any conclusions. **work-dir:** work-dir is in current workspace, prefer to use tool calls to access logs and outputs instead of custom bash commands. ### Step 3: Report Results Results format: - **Model**: `<model_id>` (`<task>`) - **Validation**: PASSED / FAILED - **Performance** (if passed): - 1st token latency, 2nd token latency, throughput - Optimum similarity / GenAI similarity (if applicable) - **Logs**: paths to export log, llm_bench log, WWB logs - **Failed step analysis** (if failed): summary of the failure and relevant log path for details ### Security - **NEVER** invoke `optimum-cli`, `wwb`, or `llm_bench` directly. Always go through `check_model.py`. - **NEVER** modify `model_id` — pass it exactly as provided by the user.