chat-test

name	chat-test
description	Run LLM inference tests with Qwen or other models. Use when testing model loading, inference, CUDA Graph, or generation quality.

LLM Chat Test

Test LLM inference with PyGPUkit.

Usage

# Basic chat CLI
python examples/chat_cli.py --model /path/to/model

# Chat with thinking mode
python examples/chat_cli_thinking.py --model /path/to/model

# MoE model (Qwen3-8B etc.)
python examples/chat_cli_moe.py --model /path/to/model

Test Models

Local test models:

• Qwen3-8B: /c/Users/y_har/.cache/huggingface/hub/models--Aratako--Qwen3-8B-ERP-v0.1/
• TinyLlama-1.1B: /c/Users/y_har/.cache/huggingface/hub/models--TinyLlama--TinyLlama-1.1B-Chat-v1.0/

Instructions

1. Ensure project is built
2. Run the appropriate chat CLI
3. Test generation quality and performance
4. Report:
   • Model loading success
   • First token latency
   • Tokens per second
   • Any errors or issues

CUDA Graph Testing

# Enable CUDA Graph for decode
python examples/chat_cli_moe.py --model /path/to/model --use-cuda-graph

Notes

• Use HuggingFace tokenizers (not built-in)
• Large models require significant VRAM
• CUDA Graph provides ~1.2x speedup for decode