菜单
Generate accurate, high-quality medical videos for clinical education and documentation by leveraging large-scale annotated medical datasets with domain-specific fine-tuning on video diffusion models.
默认使用会先检查来源的 Prompt；你也可以切换为直接命令，或下载本地副本。
决定是否安装前，请先阅读 SKILL.md，以及 SkillsMP 当前展示的配套文件。
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
直接命令不会经过审查 Prompt；运行前请先检查来源。
npx skills add https://github.com/ADu2021/skillXiv --skill medical-video-generation命令会保持在同一行。复制前请横向滚动并检查完整内容。
想先保存到本地？可下载 SkillsMP 当前能够提供的文件。
正在显示 SKILL.md
| name | medical-video-generation |
| title | MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videos |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.05675 |
| keywords | ["Medical Video","Diffusion Models","Video Synthesis","Medical AI","Domain Adaptation"] |
| description | Generate accurate, high-quality medical videos for clinical education and documentation by leveraging large-scale annotated medical datasets with domain-specific fine-tuning on video diffusion models. |
Medical video generation is critical for surgical training, patient education, and clinical documentation. However, general-purpose video generation models fail on medical content because they lack domain-specific knowledge of anatomical structures, procedural sequences, and clinical accuracy requirements. Standard video synthesis models trained on general web data cannot reliably generate medically accurate content—they may produce anatomically impossible procedures or omit critical steps.
MedGen addresses this through two key innovations: a large-scale, carefully curated medical video dataset (MedVideoCap-55K) with detailed captions, and domain-adapted fine-tuning of video diffusion models using LoRA. This approach enables generation of clinically accurate medical videos while maintaining computational efficiency, outperforming general models on medical content while matching commercial systems in quality.
Medical video generation requires training diffusion models on domain-specific data that captures the diversity of clinical contexts. Rather than collecting raw videos, the approach focuses on curation and annotation quality: filtering out artifacts, subtitles, and encoding issues; generating detailed multimodal captions describing procedures and imaging findings; and evaluating outputs through medical domain experts.
The key insight is that medical accuracy emerges from combining three elements: (1) sufficient quantity of diverse medical clips spanning clinical practice, imaging, and education, (2) rigorous quality filtering to remove technical artifacts that would confuse models, and (3) detailed text descriptions that ground video generation in clinical terminology and procedural knowledge.
The following demonstrates how to build a medical video generation system using a base diffusion model and domain-specific fine-tuning.
Step 1: Dataset Preparation and Filtering
This code applies quality filtering stages to remove unsuitable medical videos before training.
import cv2
import numpy as np
from pytesseract import pytesseract
from PIL import Image
class MedicalVideoFilter:
def __init__(self, ocr_threshold=0.3, quality_min=0.6):
self.ocr_threshold = ocr_threshold
self.quality_min = quality_min
def detect_subtitles(self, frame):
"""Use OCR to detect subtitle presence in video frames."""
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
text = pytesseract.image_to_string(gray)
# Check if substantial text detected (likely subtitles)
return len(text.split()) > self.ocr_threshold * frame.shape[0]
def assess_technical_quality(self, video_path, sample_frames=10):
"""Evaluate technical encoding quality using sharpness and noise metrics."""
cap = cv2.VideoCapture(video_path)
frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
indices = np.linspace(0, frame_count - 1, sample_frames, dtype=int)
sharpness_scores = []
for idx in indices:
cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
ret, frame = cap.read()
if ret:
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# Laplacian variance measures sharpness
sharpness = cv2.Laplacian(gray, cv2.CV_64F).var()
sharpness_scores.append(sharpness)
cap.release()
mean_sharpness = np.mean(sharpness_scores) if sharpness_scores else 0
return mean_sharpness / 1000 # Normalize to 0-1 range
def filter_batch(self, video_paths):
"""Filter videos based on subtitle presence and technical quality."""
filtered = []
for path in video_paths:
cap = cv2.VideoCapture(path)
has_subtitle = False
# Sample frames for subtitle detection
frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
for i in range(0, min(frame_count, 30), 10):
cap.set(cv2.CAP_PROP_POS_FRAMES, i)
ret, frame = cap.read()
if ret and self.detect_subtitles(frame):
has_subtitle = True
break
cap.release()
if not has_subtitle:
quality = self.assess_technical_quality(path)
if quality > self.quality_min:
filtered.append(path)
return filtered
Step 2: Caption Generation Using Multimodal LLMs
This generates detailed medical descriptions for filtered videos using multimodal language models.
from transformers import AutoProcessor, LlavaForConditionalGeneration
import torch
class MedicalCaptionGenerator:
def __init__(self, model_name="llava-1.5-7b-hf"):
self.processor = AutoProcessor.from_pretrained(model_name)
self.model = LlavaForConditionalGeneration.from_pretrained(
model_name, torch_dtype=torch.float16, device_map="auto"
)
self.device = self.model.device
def generate_caption(self, video_path, num_frames=8):
"""Extract frames and generate medical description."""
import cv2
cap = cv2.VideoCapture(video_path)
frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
frame_indices = np.linspace(0, frame_count - 1, num_frames, dtype=int)
frames = []
for idx in frame_indices:
cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
ret, frame = cap.read()
if ret:
frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))
cap.release()
# Aggregate frames into a grid for captioning
from PIL import Image as PILImage
grid = PILImage.new('RGB', (640, 480 * num_frames // 2))
for i, frame in enumerate(frames):
frame_resized = frame.resize((320, 240))
grid.paste(frame_resized, ((i % 2) * 320, (i // 2) * 240))
prompt = "Describe this medical video in clinical detail. Include: 1) anatomical structures visible, 2) medical procedures or findings, 3) clinical context."
inputs = self.processor(prompt, grid, return_tensors="pt").to(self.device)
with torch.no_grad():
output = self.model.generate(**inputs, max_new_tokens=256, temperature=0.7)
caption = self.processor.decode(output[0], skip_special_tokens=True)
return caption
Step 3: LoRA Fine-tuning for Medical Domain Adaptation
This applies parameter-efficient fine-tuning to inject medical knowledge into a base video diffusion model.
from diffusers import DiffusionPipeline
from peft import get_peft_model, LoraConfig, TaskType
import torch
from torch.utils.data import Dataset, DataLoader
class MedicalVideoDataset(Dataset):
def __init__(self, video_paths, captions, num_frames=16):
self.video_paths = video_paths
self.captions = captions
self.num_frames = num_frames
def __len__(self):
return len(self.video_paths)
def __getitem__(self, idx):
import cv2
cap = cv2.VideoCapture(self.video_paths[idx])
frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
indices = np.linspace(0, frame_count - 1, self.num_frames, dtype=int)
frames = []
for frame_idx in indices:
cap.set(cv2.CAP_PROP_POS_FRAMES, frame_idx)
ret, frame = cap.read()
if ret:
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame = torch.tensor(frame).permute(2, 0, 1).float() / 255.0
frames.append(frame)
cap.release()
frames = torch.stack(frames) if frames else torch.zeros(self.num_frames, 3, 512, 512)
caption = self.captions[idx]
return {"video": frames, "caption": caption}
class MedicalVideoFinetuner:
def __init__(self, base_model="hunyuan-video", learning_rate=1e-4):
self.pipeline = DiffusionPipeline.from_pretrained(base_model)
self.learning_rate = learning_rate
# Configure LoRA for efficient adaptation
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["to_q", "to_v"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM
)
self.model = get_peft_model(self.pipeline.unet, lora_config)
self.optimizer = torch.optim.AdamW(self.model.parameters(), lr=learning_rate)
def finetune(self, train_loader, num_epochs=3):
"""Fine-tune diffusion model on medical video data."""
self.model.train()
for epoch in range(num_epochs):
total_loss = 0
for batch in train_loader:
videos = batch["video"].to(self.model.device)
captions = batch["caption"]
# Encode text descriptions
text_embeddings = self.pipeline.encode_prompt(captions)[0]
# Add noise to videos and train denoiser
timesteps = torch.randint(0, 1000, (videos.shape[0],))
noise = torch.randn_like(videos)
noisy_videos = self.pipeline.scheduler.add_noise(videos, noise, timesteps)
# Predict noise and compute loss
pred_noise = self.model(noisy_videos, timesteps, text_embeddings).sample
loss = torch.nn.functional.mse_loss(pred_noise, noise)
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch + 1}/{num_epochs}, Loss: {total_loss / len(train_loader):.4f}")
Hyperparameters and Configuration
| Parameter | Recommended Value | Range | Notes |
|---|---|---|---|
| LoRA Rank (r) | 16-32 | 8-64 | Higher rank captures more medical specificity but increases compute |
| LoRA Alpha | 2x rank | rank-4x rank | Controls LoRA strength; 2x is conservative baseline |
| Learning Rate | 1e-4 | 1e-5 to 1e-3 | Lower rates prevent catastrophic forgetting of base knowledge |
| Num Captions per Video | 1-3 | 1-5 | Multiple angles help but diminishing returns after 3 |
| Video Frames per Clip | 16-32 | 8-64 | Higher resolution captures fine procedural details |
| Quality Filter Threshold | 0.6-0.8 | 0.4-0.9 | More aggressive filtering (0.8+) improves quality but reduces data |
When to Use
When NOT to Use
Common Pitfalls
MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videos. https://arxiv.org/abs/2507.05675
Convert arXiv papers into ready-to-use agent skills using category-aware extraction. First classifies the paper into one or more of 11 research categories, then applies a specialized extraction pipeline for each category — because different types of papers produce different types of usable knowledge. A single paper can yield multiple skills if it spans categories. Use this skill whenever the user wants to turn a paper into a skill, extract practical techniques from research, build a skill library from papers, convert arXiv papers into reusable agent instructions, or batch-process multiple papers into skills. Also trigger when someone asks about extracting actionable knowledge from papers, making research practical for LLM agents, or systematically converting academic contributions into structured agent capabilities.
Establish regret bounds for behavior cloning with discretized actions combining statistical error and quantization error terms. Prove smoothness requirements for safe quantizer design, show that learning-based quantizers fail these requirements, and propose model-based augmentation to reduce error dependence from H² to H.
Dynamically allocate LoRA ranks per-layer during fine-tuning instead of using fixed uniform ranks. Learn optimal rank for each layer and subject via variational framework with discretized exponential distribution, reducing memory footprint while maintaining fidelity and text-alignment.
基于 SOC 职业分类
