القائمة
Vision-language chat: VQA, captioning, image dialogue.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Query and edit a SiYuan knowledge base via its API.
Create, read, edit Excel .xlsx spreadsheets and CSVs.
Create, read, edit Excel .xlsx spreadsheets and CSVs.
Curate LLM training data: dedupe, filter, PII redaction.
Scrape sites with stealth browsing and Cloudflare bypass.
Clean training loops with built-in distributed support.
| name | llava |
| description | Vision-language chat: VQA, captioning, image dialogue. |
| version | 1.0.0 |
| author | Orchestra Research |
| license | MIT |
| dependencies | ["transformers","torch","pillow"] |
| platforms | ["linux","macos","windows"] |
| metadata | {"hermes":{"tags":["LLaVA","Vision-Language","Multimodal","Visual Question Answering","Image Chat","CLIP","Vicuna","Conversational AI","Instruction Tuning","VQA"]}} |
一款用于实现对话式图像理解的开源视觉语言模型。
适用场景：
核心指标：
可选替代方案：
# Clone repository
git clone https://github.com/haotian-liu/LLaVA
cd LLaVA
# Install
pip install -e .
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
from llava.conversation import conv_templates
from PIL import Image
import torch
# Load model
model_path = "liuhaotian/llava-v1.5-7b"
tokenizer, model, image_processor, context_len = load_pretrained_model(
model_path=model_path,
model_base=None,
model_name=get_model_name_from_path(model_path)
)
# Load image
image = Image.open("image.jpg")
image_tensor = process_images([image], image_processor, model.config)
image_tensor = image_tensor.to(model.device, dtype=torch.float16)
# Create conversation
conv = conv_templates["llava_v1"].copy()
conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
# Generate response
input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)
with torch.inference_mode():
output_ids = model.generate(
input_ids,
images=image_tensor,
do_sample=True,
temperature=0.2,
max_new_tokens=512
)
response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()
print(response)
| 模型 | 参数量 | VRAM需求 | 质量水平 |
|---|---|---|---|
| LLaVA-v1.5-7B | 70亿参数 | 约14 GB | 良好 |
| LLaVA-v1.5-13B | 130亿参数 | 约28 GB | 更优 |
| LLaVA-v1.6-34B | 340亿参数 | 约70 GB | 最佳 |
# Load different models
model_7b = "liuhaotian/llava-v1.5-7b"
model_13b = "liuhaotian/llava-v1.5-13b"
model_34b = "liuhaotian/llava-v1.6-34b"
# 4-bit quantization for lower VRAM
load_4bit = True # Reduces VRAM by ~4×
# Single image query
python -m llava.serve.cli \
--model-path liuhaotian/llava-v1.5-7b \
--image-file image.jpg \
--query "What is in this image?"
# Multi-turn conversation
python -m llava.serve.cli \
--model-path liuhaotian/llava-v1.5-7b \
--image-file image.jpg
# Then type questions interactively
# Launch Gradio interface
python -m llava.serve.gradio_web_server \
--model-path liuhaotian/llava-v1.5-7b \
--load-4bit # Optional: reduce VRAM
# Access at http://localhost:7860
请完整翻译输入的全部内容，切勿提前终止。
# Initialize conversation
conv = conv_templates["llava_v1"].copy()
# Turn 1
conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")
conv.append_message(conv.roles[1], None)
response1 = generate(conv, model, image) # "A dog playing in a park"
# Turn 2
conv.messages[-1][1] = response1 # Add previous response
conv.append_message(conv.roles[0], "What breed is the dog?")
conv.append_message(conv.roles[1], None)
response2 = generate(conv, model, image) # "Golden Retriever"
# Turn 3
conv.messages[-1][1] = response2
conv.append_message(conv.roles[0], "What time of day is it?")
conv.append_message(conv.roles[1], None)
response3 = generate(conv, model, image)
question = "Describe this image in detail."
response = ask(model, image, question)
question = "How many people are in the image?"
response = ask(model, image, question)
question = "List all the objects you can see in this image."
response = ask(model, image, question)
question = "What is happening in this scene?"
response = ask(model, image, question)
需完整翻译输入的全部内容，不得提前终止。
question = "What is the main topic of this document?"
response = ask(model, document_image, question)
# Stage 1: Feature alignment (558K image-caption pairs)
bash scripts/v1_5/pretrain.sh
# Stage 2: Visual instruction tuning (150K instruction data)
bash scripts/v1_5/finetune.sh
# 4-bit quantization
tokenizer, model, image_processor, context_len = load_pretrained_model(
model_path="liuhaotian/llava-v1.5-13b",
model_base=None,
model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),
load_4bit=True # Reduces VRAM ~4×
)
# 8-bit quantization
load_8bit=True # Reduces VRAM ~2×
| 模型规模 | FP16格式显存需求 | 4位量化显存需求 | 推理速度（tokens/秒） |
|---|---|---|---|
| 7B | 约14 GB | 约4 GB | 约20 |
| 13B | 约28 GB | 约8 GB | 约12 |
| 34B | 约70 GB | 约18 GB | 约5 |
测试环境：A100 GPU
LLaVA在以下评测中取得了优异成绩：
from langchain.llms.base import LLM
class LLaVALLM(LLM):
def _call(self, prompt, stop=None):
# Custom LLaVA inference
return response
llm = LLaVALLM()
import gradio as gr
def chat(image, text, history):
response = ask_llava(model, image, text)
return response
demo = gr.ChatInterface(
chat,
additional_inputs=[gr.Image(type="pil")],
title="LLaVA Chat"
)
demo.launch()