tars-token-adaptive-hallucination-reduction

Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding.

Zur Installation springen

Repository: ADu2021/skillXiv
Letzte Quellaktivität: 24. März 2026 um 19:42
Erkannte Sprache von SKILL.md: Englisch
Sterne: 6
Forks: 0

Installationsoptionen

Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.

Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.

Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.

npx skills add https://github.com/ADu2021/skillXiv --skill tars-token-adaptive-hallucination-reduction

Der Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.

Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.

Quelldateien prüfen

Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.

SKILL.md

Quellanweisungen · Schreibgeschützte Vorschau

Zurück zur Suche Zurück zur Installation

import torch import torch.nn.functional as F from typing import List, Tuple, Dict class PreferenceEncoder: """Encodes human preferences as token-level distributions""" def __init__(): .tokenizer = tokenizer vocab_size = vocabulary_size () -> [torch.Tensor, torch.Tensor]: preferred_tokens = .tokenizer.encode(preferred_text) dispreferred_tokens = .tokenizer.encode(dispreferred_text) max_len = ((preferred_tokens), (dispreferred_tokens)) preferred_dist = torch.zeros(max_len, .vocab_size) dispreferred_dist = torch.zeros(max_len, .vocab_size) i, token_id (preferred_tokens): preferred_dist[i, token_id] = i, token_id (dispreferred_tokens): dispreferred_dist[i, token_id] = preferred_dist, dispreferred_dist () -> torch.Tensor: all_prefs = [] preferred, dispreferred pairs: pref_dist, _ = .encode_preference_pair(preferred, dispreferred) all_prefs.append(pref_dist) max_len = (p.shape[] p all_prefs) batch_prefs = torch.zeros((all_prefs), max_len, .vocab_size) i, pref (all_prefs): batch_prefs[i, :pref.shape[], :] = pref batch_prefs

class MinMaxPreferenceOptimizer: """Min-max optimization for robust preference learning""" def __init__(self, model, tokenizer, epsilon: float = 0.1): self.model = model self.tokenizer = tokenizer self.epsilon = epsilon # Perturbation budget self.vocab_size = model.config.vocab_size def compute_preference_loss(self, logits: torch.Tensor, target_dist: torch.Tensor, positions: torch.Tensor) -> torch.Tensor: """ Compute loss between predicted logits and preference targets. Args: logits: Model output logits, shape (batch, seq_len, vocab_size) target_dist: Target probability distribution over tokens positions: Which positions correspond to preference targets Returns: Scalar loss value """ # Extract logits at preference positions pred_probs = F.softmax(logits, dim=-1) # KL divergence from model distribution to target loss = F.kl_div( F.log_softmax(logits, dim=-1), target_dist.clamp(min=1e-8), reduction='batchmean' ) return loss def perturbation_step(self, target_dist: torch.Tensor, step_size: float = 0.01) -> torch.Tensor: """ Inner optimization: find worst-case perturbation within semantic budget. Args: target_dist: Current preference target step_size: Optimization step size Returns: Perturbed distribution that remains semantically valid """ perturbed = target_dist.clone().detach().requires_grad_(True) optimizer = torch.optim.Adam([perturbed], lr=step_size) for _ in range(5): # Inner loop iterations # Compute model loss on perturbed targets logits = self.model(...) # Forward pass loss = self.compute_preference_loss(logits, perturbed, mask=None) # Maximize loss (adversarial): find worst perturbation loss_adv = -loss optimizer.zero_grad() loss_adv.backward() optimizer.step() # Project back to semantic constraint set perturbed.data = self._project_to_semantic_budget(perturbed.data) return perturbed.detach() def _project_to_semantic_budget(self, dist: torch.Tensor) -> torch.Tensor: """ Project distribution back to semantic constraint set. Keep similar tokens with higher weight. """ # Reproject to valid probability distribution dist = F.softmax(dist, dim=-1) # Constraint: perturbed distribution cannot exceed epsilon away from original # (in terms of TV divergence or other semantic distance metric) # Simplified: clamp large changes return dist def min_max_training_step(self, images: torch.Tensor, prompts: torch.Tensor, preference_targets: torch.Tensor) -> Tuple[torch.Tensor, float]: """ Single min-max training step: maximize robustness to preference perturbations. Args: images: Visual input prompts: Text prompts preference_targets: Target distributions for next tokens Returns: (updated_model_state, loss_value) """ # Outer loop: optimize model parameters optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5) for outer_step in range(3): # Outer iterations # Inner loop: find worst perturbation perturbed_targets = self.perturbation_step(preference_targets) # Forward pass with model outputs = self.model(images, prompts) logits = outputs.logits # Compute loss on perturbed targets loss = self.compute_preference_loss(logits, perturbed_targets, positions=None) # Outer optimization: minimize loss on worst-case perturbation optimizer.zero_grad() loss.backward() optimizer.step() return self.model.state_dict(), loss.item()

class TokenAdaptivePreference: """Adaptively applies preference steering per token""" def __init__(self, model): self.model = model def compute_token_importance(self, logits: torch.Tensor, target_dist: torch.Tensor) -> torch.Tensor: """ Compute importance weight for each token based on preference strength. Args: logits: Model predictions target_dist: Preference targets Returns: Importance weights of shape (batch, seq_len) """ pred_probs = F.softmax(logits, dim=-1) # Importance = entropy difference (high-entropy targets need stronger steering) target_entropy = -(target_dist * torch.log(target_dist + 1e-8)).sum(dim=-1) pred_entropy = -(pred_probs * torch.log(pred_probs + 1e-8)).sum(dim=-1) importance = (target_entropy - pred_entropy).clamp(min=0) # Normalize to [0, 1] importance = importance / (importance.max() + 1e-8) return importance def apply_adaptive_preference(self, logits: torch.Tensor, target_dist: torch.Tensor, base_strength: float = 1.0) -> torch.Tensor: """ Shift logits toward preference targets with token-adaptive strength. Args: logits: Model logits target_dist: Preference targets base_strength: Base preference steering strength Returns: Modified logits """ # Compute per-token importance weights importance = self.compute_token_importance(logits, target_dist) # Compute preference direction pred_probs = F.softmax(logits, dim=-1) pref_direction = target_dist - pred_probs # Shape: (batch, seq_len, vocab) # Apply adaptive steering batch_size, seq_len, vocab_size = logits.shape for b in range(batch_size): for t in range(seq_len): # Scale steering by token importance strength = base_strength * importance[b, t].item() # Move logits toward preference targets logits[b, t, :] = logits[b, t, :] + strength * pref_direction[b, t, :] return logits

class VisualGroundingPreserver: """Ensures preference optimization preserves visual grounding""" def __init__(self, model, vision_encoder): self.model = model self.vision_encoder = vision_encoder def compute_visual_consistency(self, image: torch.Tensor, output_tokens: torch.Tensor) -> float: """ Measure how well output is grounded in visual content. Compute correlation between image features and output token embeddings. """ # Extract visual features image_features = self.vision_encoder(image) # (image_dim,) # Extract output token embeddings token_embeddings = self.model.get_token_embeddings(output_tokens) # (seq_len, emb_dim) # Compute attention-weighted consistency batch_size, seq_len, vocab_size = output_tokens.shape similarities = [] for t in range(seq_len): # Similarity between this token's embedding and image features token_emb = token_embeddings[t] similarity = F.cosine_similarity( image_features.unsqueeze(0), token_emb.unsqueeze(0) ) similarities.append(similarity) # Average similarity across tokens visual_consistency = torch.mean(torch.stack(similarities)) return visual_consistency.item() def grounding_aware_loss(self, logits: torch.Tensor, target_dist: torch.Tensor, image: torch.Tensor, lambda_grounding: float = 0.1) -> torch.Tensor: """ Combine preference loss with visual grounding preservation. Args: logits: Model predictions target_dist: Preference targets image: Visual input lambda_grounding: Weight for grounding constraint Returns: Total loss that preserves grounding """ # Preference alignment loss preference_loss = F.kl_div( F.log_softmax(logits, dim=-1), target_dist, reduction='batchmean' ) # Visual grounding penalty predicted_tokens = torch.argmax(logits, dim=-1) grounding_score = self.compute_visual_consistency(image, predicted_tokens) # Total loss: preference optimization + grounding preservation total_loss = preference_loss - lambda_grounding * grounding_score return total_loss

class TARSTrainer: """Complete TARS training with preference, perturbation, and grounding""" def __init__(self, model, vision_encoder, tokenizer, epsilon: float = 0.1, lambda_grounding: float = 0.1): self.model = model self.vision_encoder = vision_encoder self.pref_encoder = PreferenceEncoder(tokenizer, model.config.vocab_size) self.optimizer = MinMaxPreferenceOptimizer(model, tokenizer, epsilon) self.token_adaptive = TokenAdaptivePreference(model) self.grounding = VisualGroundingPreserver(model, vision_encoder) self.lambda_grounding = lambda_grounding def train_step(self, batch: Dict) -> float: """ Single TARS training step. Args: batch: Contains 'images', 'prompts', 'preferred_text', 'dispreferred_text' Returns: Loss value """ images = batch['images'] prompts = batch['prompts'] preferred = batch['preferred_text'] dispreferred = batch['dispreferred_text'] # Encode preferences as distributions pref_targets = self.pref_encoder.create_preference_labels( list(zip(preferred, dispreferred)) ) # Forward pass outputs = self.model(images, prompts) logits = outputs.logits # Apply token-adaptive preference steering adapted_logits = self.token_adaptive.apply_adaptive_preference( logits, pref_targets, base_strength=1.0 ) # Compute loss with visual grounding preservation loss = self.grounding.grounding_aware_loss( adapted_logits, pref_targets, images, lambda_grounding=self.lambda_grounding ) # Min-max optimization step self.model.train() loss.backward() torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0) optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5) optimizer.step() optimizer.zero_grad() return loss.item() def train_epoch(self, dataloader, num_epochs: int = 3): """Train for multiple epochs""" for epoch in range(num_epochs): total_loss = 0.0 for batch_idx, batch in enumerate(dataloader): loss = self.train_step(batch) total_loss += loss if batch_idx % 100 == 0: print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss:.4f}") avg_loss = total_loss / len(dataloader) print(f"Epoch {epoch} average loss: {avg_loss:.4f}")

name	tars-token-adaptive-hallucination-reduction
title	TARS Min-Max Token-Adaptive Preference Strategy
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.21584
keywords	["multimodal-llm","hallucination-reduction","preference-optimization","token-adaptive","robustness"]
description	Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding.

name	tars-token-adaptive-hallucination-reduction
title	TARS Min-Max Token-Adaptive Preference Strategy
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.21584
keywords	["multimodal-llm","hallucination-reduction","preference-optimization","token-adaptive","robustness"]
description	Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding.

tars-token-adaptive-hallucination-reduction

Installationsoptionen

Quelldateien prüfen

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

Practical Guidance

Reference

tars-token-adaptive-hallucination-reduction

Quellinformationen

Installationsoptionen

Quelldateien prüfen

Mehr aus diesem Repository

Mehr aus diesem Repository

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

Practical Guidance

Reference