메뉴
Use when "CLIP", "Whisper", "Stable Diffusion", "SDXL", "speech-to-text", "text-to-image", "image generation", "transcription", "zero-shot classification", "image-text similarity", "inpainting", "ControlNet"
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | multimodal-models |
| description | Use when "CLIP", "Whisper", "Stable Diffusion", "SDXL", "speech-to-text", "text-to-image", "image generation", "transcription", "zero-shot classification", "image-text similarity", "inpainting", "ControlNet" |
| version | 1.0.0 |
Pre-trained models for vision, audio, and cross-modal tasks.
| Model | Modality | Task |
|---|---|---|
| CLIP | Image + Text | Zero-shot classification, similarity |
| Whisper | Audio → Text | Transcription, translation |
| Stable Diffusion | Text → Image | Image generation, editing |
Zero-shot image classification without training on specific labels.
| Task | How |
|---|---|
| Zero-shot classification | Compare image to text label embeddings |
| Image search | Find images matching text query |
| Content moderation | Classify against safety categories |
| Image similarity | Compare image embeddings |
| Model | Parameters | Trade-off |
|---|---|---|
| ViT-B/32 | 151M | Recommended balance |
| ViT-L/14 | 428M | Best quality, slower |
| RN50 | 102M | Fastest, lower quality |
| Concept | Description |
|---|---|
| Dual encoder | Separate encoders for image and text |
| Contrastive learning | Trained to match image-text pairs |
| Normalization | Always normalize embeddings before similarity |
| Descriptive labels | Better labels = better zero-shot accuracy |
Key concept: CLIP embeds images and text in same space. Classification = find nearest text embedding.
Robust multilingual transcription supporting 99 languages.
| Task | Configuration |
|---|---|
| Transcription | Default transcribe task |
| Translation to English | task="translate" |
| Subtitles | Output format SRT/VTT |
| Word timestamps | word_timestamps=True |
| Model | Size | Speed | Recommendation |
|---|---|---|---|
| turbo | 809M | Fast | Recommended |
| large | 1550M | Slow | Maximum quality |
| small | 244M | Medium | Good balance |
| base | 74M | Fast | Quick tests |
| tiny | 39M | Fastest | Prototyping only |
| Concept | Description |
|---|---|
| Language detection | Auto-detects, or specify for speed |
| Initial prompt | Improves technical terms accuracy |
| Timestamps | Segment-level or word-level |
| faster-whisper | 4× faster alternative implementation |
Key concept: Specify language when known—auto-detection adds latency.
Text-to-image generation with various control methods.
| Task | Pipeline |
|---|---|
| Text-to-image | DiffusionPipeline |
| Style transfer | Image2Image |
| Fill regions | Inpainting |
| Guided generation | ControlNet |
| Custom styles | LoRA adapters |
| Model | Resolution | Quality |
|---|---|---|
| SDXL | 1024×1024 | Best |
| SD 1.5 | 512×512 | Good, faster |
| SD 2.1 | 768×768 | Middle ground |
| Parameter | Effect | Typical Value |
|---|---|---|
| num_inference_steps | Quality vs speed | 20-50 |
| guidance_scale | Prompt adherence | 7-12 |
| negative_prompt | Avoid artifacts | "blurry, low quality" |
| strength (img2img) | How much to change | 0.5-0.8 |
| seed | Reproducibility | Fixed number |
| Method | Input | Use Case |
|---|---|---|
| ControlNet | Edge/depth/pose | Structural guidance |
| LoRA | Trained weights | Custom styles |
| Img2Img | Source image | Style transfer |
| Inpainting | Image + mask | Fill regions |
| Technique | Effect |
|---|---|
| CPU offload | Reduces VRAM usage |
| Attention slicing | Trades speed for memory |
| VAE tiling | Large image support |
| xFormers | Faster attention |
| DPM scheduler | Fewer steps needed |
Key concept: Use SDXL for quality, SD 1.5 for speed. Always use negative prompts.
All three models use embeddings:
| Model | VRAM Needed |
|---|---|
| CLIP ViT-B/32 | ~2 GB |
| Whisper turbo | ~6 GB |
| SD 1.5 | ~6 GB |
| SDXL | ~10 GB |
| Practice | Why |
|---|---|
| Use recommended model sizes | Best quality/speed balance |
| Cache embeddings (CLIP) | Expensive to recompute |
| Specify language (Whisper) | Faster than auto-detect |
| Use negative prompts (SD) | Avoid common artifacts |
| Set seeds for reproducibility | Consistent results |
Use when evaluating agent performance, building test frameworks, measuring quality, or asking about "agent evaluation", "LLM-as-judge", "agent testing", "quality metrics", "evaluation rubrics", "agent benchmarks"
Use when "codex", "use gpt", "gpt-5", "openai codex", "let openai", "full-auto", "autonomous code generation"
Use when compressing agent context, implementing conversation summarization, reducing token usage in long sessions, or asking about "context compression", "conversation history", "token optimization", "context limits", "summarization strategies"
Use when diagnosing agent failures, debugging lost-in-middle issues, understanding context poisoning, or asking about "context degradation", "lost in middle", "context poisoning", "attention patterns", "context clash", "agent performance drops"
Use when optimizing agent context, reducing token costs, implementing KV-cache optimization, or asking about "context optimization", "token reduction", "context limits", "observation masking", "context budgeting", "context partitioning"
Use when "CrewAI", "multi-agent systems", "agent orchestration", "AI crews", or asking about "autonomous agents", "agent collaboration", "role-based agents", "agent workflows", "AI team coordination"
