llava

name	llava
description	Large Language and Vision Assistant. Enables visual instruction tuning and image-based conversations. Combines CLIP vision encoder with Vicuna/LLaMA language models. Supports multi-turn image chat, visual question answering, and instruction following. Use for vision-language chatbots or image understanding tasks. Best for conversational image analysis.
version	1.0.0
author	Orchestra Research
license	MIT
dependencies	["transformers","torch","pillow"]
platforms	["linux","macos","windows"]
metadata	{"sonic":{"tags":["LLaVA","Vision-Language","Multimodal","Visual Question Answering","Image Chat","CLIP","Vicuna","Conversational AI","Instruction Tuning","VQA"]}}

LLaVA - Large Language and Vision Assistant

Open-source vision-language model for conversational image understanding.

When to use LLaVA

Use when:

• Building vision-language chatbots
• Visual question answering (VQA)
• Image description and captioning
• Multi-turn image conversations
• Visual instruction following
• Document understanding with images

Metrics:

• 23,000+ GitHub stars
• GPT-4V level capabilities (targeted)
• Apache 2.0 License
• Multiple model sizes (7B-34B params)

Use alternatives instead:

• GPT-4V: Highest quality, API-based
• CLIP: Simple zero-shot classification
• BLIP-2: Better for captioning only
• Flamingo: Research, not open-source

Quick start

Installation

# Clone repository
git clone https://github.com/haotian-liu/LLaVA
cd LLaVA

# Install
pip install -e .

Basic usage

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.conversation import conv_templates
from PIL import Image
import torch

# Load model
model_path = "liuhaotian/llava-v1.5-7b"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)

# Load image
image = Image.open("image.jpg")
image_tensor = process_images([image], image_processor, model.config)
image_tensor = image_tensor.to(model.device, dtype=torch.float16)


conv = conv_templates[].copy()
conv.append_message(conv.roles[], DEFAULT_IMAGE_TOKEN + )
conv.append_message(conv.roles[], )
prompt = conv.get_prompt()


input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors=).unsqueeze().to(model.device)

 torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        images=image_tensor,
        do_sample=,
        temperature=,
        max_new_tokens=
    )

response = tokenizer.decode(output_ids[], skip_special_tokens=).strip()
(response)