model-researcher

Investigate model specifications, requirements, and evaluation strategy. Use when researching a model before benchmarking: reading HuggingFace model cards, estimating VRAM requirements, selecting GPU providers, and determining evaluation approach. Triggers on "model research", "investigate model", "model info", "VRAM estimate", "which provider", "model card".

Jump to install

Repository: nyosegawa/agentic-bench
Last source activity: February 21, 2026 at 07:23
Detected SKILL.md language: English
Stars: 5
Forks: 0

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

Back to search Back to installation

name	model-researcher
description	Investigate model specifications, requirements, and evaluation strategy. Use when researching a model before benchmarking: reading HuggingFace model cards, estimating VRAM requirements, selecting GPU providers, and determining evaluation approach. Triggers on "model research", "investigate model", "model info", "VRAM estimate", "which provider", "model card".

Model Researcher

You are an experienced ML engineer investigating a model before running benchmarks.

Your Goal

Given a model name, produce a structured research summary:

Model identity (type, architecture, parameter count, license)
Hardware requirements (VRAM, recommended GPU)
Recommended provider (cheapest viable option)
Evaluation strategy (what to test and how)

Workflow

Step 1: Gather Model Information

Run the helper script to pull structured metadata:

python .claude/skills/model-researcher/scripts/hf_model_info.py MODEL_ID

To check if the model is available on HF Inference API (serverless):

python .claude/skills/model-researcher/scripts/hf_inference_check.py MODEL_ID

To search for similar or alternative models:

python .claude/skills/model-researcher/scripts/hf_model_search.py --task llm --sort downloads --limit 10
python .claude/skills/model-researcher/scripts/hf_model_search.py --search "qwen" --limit 5

If the script fails or the model is not on HuggingFace:

Search the web for the model's official page, paper, or GitHub repo
Manually gather: architecture, parameter count, input/output modalities, license

Step 2: Estimate VRAM and Cost

python .claude/skills/model-researcher/scripts/gpu_estimator.py \
  --params PARAM_COUNT --quant fp16 --model-type llm --check-env --json

Model Type	Reference File	Key Metrics
LLM (text-to-text)	`references/eval-llm.md`	tokens/sec, latency, output quality
VLM (vision-language)	`references/eval-vlm.md`	tokens/sec, hallucination rate, OCR accuracy
Code Generation	`references/eval-code-gen.md`	tokens/sec, pass@1, FIM accuracy
Embedding	`references/eval-embedding.md`	embeddings/sec, retrieval quality
Image Generation	`references/eval-image.md`	sec/image, visual quality
TTS (text-to-speech)	`references/eval-tts.md`	RTF, audio quality
STT (speech recognition)	`references/eval-stt.md`	RTF, WER/CER
Audio / Music Gen	`references/eval-audio.md`	RTF, sec/generation, listening quality
Video Generation	`references/eval-video-gen.md`	sec/frame, temporal consistency
Object Detection	`references/eval-object-detection.md`	mAP, FPS, detection quality
3D Generation	`references/eval-3d-gen.md`	generation time, mesh quality
Time Series	`references/eval-timeseries.md`	MAE/RMSE, prediction accuracy

## Research Summary: {model_name}
- **URL**: https://huggingface.co/{model_id}
- **Description**: {1-2 sentence summary from model card}
- **Type**: {model_type}
- **Parameters**: {param_count}
- **Architecture**: {architecture}
- **License**: {license}
- **Notable Features**: {e.g., "6 language tags for accent switching", "supports img2img"}
- **Install**: {pip install command from model card, if available}
- **VRAM Required**: {vram_estimate}
- **Recommended GPU**: {gpu}
- **Recommended Provider**: {provider}
- **Estimated Cost**: {cost} (e.g., "~$0.53 on Modal A100-40GB, ~15min" or "Free via HF Inference API")
- **Evaluation Strategy**: {brief description}
- **Key Metrics**: {metrics to measure}