tars-token-adaptive-hallucination-reduction

Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding.

Jump to install

Repository: ADu2021/skillXiv
Last source activity: March 24, 2026 at 19:42
Detected SKILL.md language: English
Stars: 6
Forks: 0

Install options

The review-first prompt is selected by default. You can switch to a direct command or download a local copy.

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

A direct command skips the review prompt. Inspect the source before running it.

npx skills add https://github.com/ADu2021/skillXiv --skill tars-token-adaptive-hallucination-reduction

The command stays on one line. Scroll horizontally to inspect it before copying.

Prefer a local copy? Download the files currently available to SkillsMP.

Review the source files

Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.

SKILL.md

Source instructions · Read-only preview

Back to search Back to installation

import torch import torch.nn.functional as F from typing import List, Tuple, Dict class PreferenceEncoder: """Encodes human preferences as token-level distributions""" def __init__(): .tokenizer = tokenizer vocab_size = vocabulary_size () -> [torch.Tensor, torch.Tensor]: preferred_tokens = .tokenizer.encode(preferred_text) dispreferred_tokens = .tokenizer.encode(dispreferred_text) max_len = ((preferred_tokens), (dispreferred_tokens)) preferred_dist = torch.zeros(max_len, .vocab_size) dispreferred_dist = torch.zeros(max_len, .vocab_size) i, token_id (preferred_tokens): preferred_dist[i, token_id] = i, token_id (dispreferred_tokens): dispreferred_dist[i, token_id] = preferred_dist, dispreferred_dist () -> torch.Tensor: all_prefs = [] preferred, dispreferred pairs: pref_dist, _ = .encode_preference_pair(preferred, dispreferred) all_prefs.append(pref_dist) max_len = (p.shape[] p all_prefs) batch_prefs = torch.zeros((all_prefs), max_len, .vocab_size) i, pref (all_prefs): batch_prefs[i, :pref.shape[], :] = pref batch_prefs

class MinMaxPreferenceOptimizer: """Min-max optimization for robust preference learning""" def __init__(self, model, tokenizer, epsilon: float = 0.1): self.model = model self.tokenizer = tokenizer self.epsilon = epsilon # Perturbation budget self.vocab_size = model.config.vocab_size def compute_preference_loss(self, logits: torch.Tensor, target_dist: torch.Tensor, positions: torch.Tensor) -> torch.Tensor: """ Compute loss between predicted logits and preference targets. Args: logits: Model output logits, shape (batch, seq_len, vocab_size) target_dist: Target probability distribution over tokens positions: Which positions correspond to preference targets Returns: Scalar loss value """ # Extract logits at preference positions pred_probs = F.softmax(logits, dim=-1) # KL divergence from model distribution to target loss = F.kl_div( F.log_softmax(logits, dim=-1), target_dist.clamp(min=1e-8), reduction='batchmean' ) return loss def perturbation_step(self, target_dist: torch.Tensor, step_size: float = 0.01) -> torch.Tensor: """ Inner optimization: find worst-case perturbation within semantic budget. Args: target_dist: Current preference target step_size: Optimization step size Returns: Perturbed distribution that remains semantically valid """ perturbed = target_dist.clone().detach().requires_grad_(True) optimizer = torch.optim.Adam([perturbed], lr=step_size) for _ in range(5): # Inner loop iterations # Compute model loss on perturbed targets logits = self.model(...) # Forward pass loss = self.compute_preference_loss(logits, perturbed, mask=None) # Maximize loss (adversarial): find worst perturbation loss_adv = -loss optimizer.zero_grad() loss_adv.backward() optimizer.step() # Project back to semantic constraint set perturbed.data = self._project_to_semantic_budget(perturbed.data) return perturbed.detach() def _project_to_semantic_budget(self, dist: torch.Tensor) -> torch.Tensor: """ Project distribution back to semantic constraint set. Keep similar tokens with higher weight. """ # Reproject to valid probability distribution dist = F.softmax(dist, dim=-1) # Constraint: perturbed distribution cannot exceed epsilon away from original # (in terms of TV divergence or other semantic distance metric) # Simplified: clamp large changes return dist def min_max_training_step(self, images: torch.Tensor, prompts: torch.Tensor, preference_targets: torch.Tensor) -> Tuple[torch.Tensor, float]: """ Single min-max training step: maximize robustness to preference perturbations. Args: images: Visual input prompts: Text prompts preference_targets: Target distributions for next tokens Returns: (updated_model_state, loss_value) """ # Outer loop: optimize model parameters optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5) for outer_step in range(3): # Outer iterations # Inner loop: find worst perturbation perturbed_targets = self.perturbation_step(preference_targets) # Forward pass with model outputs = self.model(images, prompts) logits = outputs.logits # Compute loss on perturbed targets loss = self.compute_preference_loss(logits, perturbed_targets, positions=None) # Outer optimization: minimize loss on worst-case perturbation optimizer.zero_grad() loss.backward() optimizer.step() return self.model.state_dict(), loss.item()

class TokenAdaptivePreference: """Adaptively applies preference steering per token""" def __init__(self, model): self.model = model def compute_token_importance(self, logits: torch.Tensor, target_dist: torch.Tensor) -> torch.Tensor: """ Compute importance weight for each token based on preference strength. Args: logits: Model predictions target_dist: Preference targets Returns: Importance weights of shape (batch, seq_len) """ pred_probs = F.softmax(logits, dim=-1) # Importance = entropy difference (high-entropy targets need stronger steering) target_entropy = -(target_dist * torch.log(target_dist + 1e-8)).sum(dim=-1) pred_entropy = -(pred_probs * torch.log(pred_probs + 1e-8)).sum(dim=-1) importance = (target_entropy - pred_entropy).clamp(min=0) # Normalize to [0, 1] importance = importance / (importance.max() + 1e-8) return importance def apply_adaptive_preference(self, logits: torch.Tensor, target_dist: torch.Tensor, base_strength: float = 1.0) -> torch.Tensor: """ Shift logits toward preference targets with token-adaptive strength. Args: logits: Model logits target_dist: Preference targets base_strength: Base preference steering strength Returns: Modified logits """ # Compute per-token importance weights importance = self.compute_token_importance(logits, target_dist) # Compute preference direction pred_probs = F.softmax(logits, dim=-1) pref_direction = target_dist - pred_probs # Shape: (batch, seq_len, vocab) # Apply adaptive steering batch_size, seq_len, vocab_size = logits.shape for b in range(batch_size): for t in range(seq_len): # Scale steering by token importance strength = base_strength * importance[b, t].item() # Move logits toward preference targets logits[b, t, :] = logits[b, t, :] + strength * pref_direction[b, t, :] return logits

class VisualGroundingPreserver: """Ensures preference optimization preserves visual grounding""" def __init__(self, model, vision_encoder): self.model = model self.vision_encoder = vision_encoder def compute_visual_consistency(self, image: torch.Tensor, output_tokens: torch.Tensor) -> float: """ Measure how well output is grounded in visual content. Compute correlation between image features and output token embeddings. """ # Extract visual features image_features = self.vision_encoder(image) # (image_dim,) # Extract output token embeddings token_embeddings = self.model.get_token_embeddings(output_tokens) # (seq_len, emb_dim) # Compute attention-weighted consistency batch_size, seq_len, vocab_size = output_tokens.shape similarities = [] for t in range(seq_len): # Similarity between this token's embedding and image features token_emb = token_embeddings[t] similarity = F.cosine_similarity( image_features.unsqueeze(0), token_emb.unsqueeze(0) ) similarities.append(similarity) # Average similarity across tokens visual_consistency = torch.mean(torch.stack(similarities)) return visual_consistency.item() def grounding_aware_loss(self, logits: torch.Tensor, target_dist: torch.Tensor, image: torch.Tensor, lambda_grounding: float = 0.1) -> torch.Tensor: """ Combine preference loss with visual grounding preservation. Args: logits: Model predictions target_dist: Preference targets image: Visual input lambda_grounding: Weight for grounding constraint Returns: Total loss that preserves grounding """ # Preference alignment loss preference_loss = F.kl_div( F.log_softmax(logits, dim=-1), target_dist, reduction='batchmean' ) # Visual grounding penalty predicted_tokens = torch.argmax(logits, dim=-1) grounding_score = self.compute_visual_consistency(image, predicted_tokens) # Total loss: preference optimization + grounding preservation total_loss = preference_loss - lambda_grounding * grounding_score return total_loss

class TARSTrainer: """Complete TARS training with preference, perturbation, and grounding""" def __init__(self, model, vision_encoder, tokenizer, epsilon: float = 0.1, lambda_grounding: float = 0.1): self.model = model self.vision_encoder = vision_encoder self.pref_encoder = PreferenceEncoder(tokenizer, model.config.vocab_size) self.optimizer = MinMaxPreferenceOptimizer(model, tokenizer, epsilon) self.token_adaptive = TokenAdaptivePreference(model) self.grounding = VisualGroundingPreserver(model, vision_encoder) self.lambda_grounding = lambda_grounding def train_step(self, batch: Dict) -> float: """ Single TARS training step. Args: batch: Contains 'images', 'prompts', 'preferred_text', 'dispreferred_text' Returns: Loss value """ images = batch['images'] prompts = batch['prompts'] preferred = batch['preferred_text'] dispreferred = batch['dispreferred_text'] # Encode preferences as distributions pref_targets = self.pref_encoder.create_preference_labels( list(zip(preferred, dispreferred)) ) # Forward pass outputs = self.model(images, prompts) logits = outputs.logits # Apply token-adaptive preference steering adapted_logits = self.token_adaptive.apply_adaptive_preference( logits, pref_targets, base_strength=1.0 ) # Compute loss with visual grounding preservation loss = self.grounding.grounding_aware_loss( adapted_logits, pref_targets, images, lambda_grounding=self.lambda_grounding ) # Min-max optimization step self.model.train() loss.backward() torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0) optimizer = torch.optim.AdamW(self.model.parameters(), lr=1e-5) optimizer.step() optimizer.zero_grad() return loss.item() def train_epoch(self, dataloader, num_epochs: int = 3): """Train for multiple epochs""" for epoch in range(num_epochs): total_loss = 0.0 for batch_idx, batch in enumerate(dataloader): loss = self.train_step(batch) total_loss += loss if batch_idx % 100 == 0: print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss:.4f}") avg_loss = total_loss / len(dataloader) print(f"Epoch {epoch} average loss: {avg_loss:.4f}")

name	tars-token-adaptive-hallucination-reduction
title	TARS Min-Max Token-Adaptive Preference Strategy
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.21584
keywords	["multimodal-llm","hallucination-reduction","preference-optimization","token-adaptive","robustness"]
description	Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding.

name	tars-token-adaptive-hallucination-reduction
title	TARS Min-Max Token-Adaptive Preference Strategy
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2507.21584
keywords	["multimodal-llm","hallucination-reduction","preference-optimization","token-adaptive","robustness"]
description	Token-adaptive preference optimization framework using min-max formulation to reduce multimodal LLM hallucination. Achieves 50% hallucination reduction using min-max distributional robustness while preserving visual grounding.

tars-token-adaptive-hallucination-reduction

Install options

Review the source files

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

Practical Guidance

Reference

tars-token-adaptive-hallucination-reduction

Source facts

Install options

Review the source files

More from this repository

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction

Core Concept

Architecture Overview

Implementation Steps

Practical Guidance

Reference

More from this repository