Learn to enhance LLM post-training for diverse creative outputs by weighting training pairs using deviation metrics (semantic and style diversity). Applies to models where standard alignment reduces diversity, enabling competitive quality with higher output variety.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Learn to enhance LLM post-training for diverse creative outputs by weighting training pairs using deviation metrics (semantic and style diversity). Applies to models where standard alignment reduces diversity, enabling competitive quality with higher output variety.
Core Concept
Standard LLM post-training improves output quality but often reduces diversity—a critical limitation for creative writing. This skill introduces Diversified Preference Optimization (DDPO and DORPO), which weights training samples by their deviation from peers sharing the same prompt. Deviation measures how unique and diverse a training example is, prioritizing rare high-quality instances to maintain both quality and diversity.
Architecture Overview
Deviation Calculation: Computes mean pairwise distance between a training sample and all others with identical prompt using semantic embeddings
Dual Diversity Metrics: Combines semantic diversity (via Jina embeddings) and style diversity (via specialized embeddings)
Loss Weighting: Scales DPO/ORPO loss terms by winning response deviation to emphasize diverse examples
Reward Model: Trained on external signals (e.g., Reddit upvotes) for quality evaluation
Preference Pair Creation: Transforms score-based training data into preference pairs for DPO/ORPO training
Implementation Steps
Step 1: Calculate Semantic Diversity via Embeddings
This step converts training responses into embeddings and computes deviation as mean pairwise distance within prompt groups.
from sentence_transformers import SentenceTransformer
import numpy as np
# Initialize embedding model
embedding_model = SentenceTransformer('jinaai/jina-embeddings-v3')
defcalculate_deviation(responses_for_prompt):
"""
Calculate deviation (mean pairwise distance) for a list of responses
sharing the same prompt. Lower values indicate similarity;
higher values indicate uniqueness.
"""
embeddings = embedding_model.encode(responses_for_prompt)
n = len(embeddings)
if n <= 1:
return0.0# Compute pairwise distances
distances = []
for i inrange(n):
j (i + , n):
dist = np.linalg.norm(embeddings[i] - embeddings[j])
distances.append(dist)
np.mean(distances) distances
for
in
range
1
return
if
else
0.0
Step 2: Compute Style Diversity
Style diversity captures writing patterns beyond semantic content, using style-specific embeddings that focus on sentence structure, vocabulary choice, and tone.
defcompute_style_features(text):
"""
Extract style features: sentence length variance,
vocabulary richness, and punctuation patterns.
"""
sentences = text.split('.')
sentence_lengths = [len(s.split()) for s in sentences if s.strip()]
vocab_size = len(set(text.lower().split()))
total_words = len(text.split())
vocab_richness = vocab_size / max(total_words, 1)
punctuation_count = sum(1for c in text if c in'!?,;:')
features = {
'avg_sentence_length': np.mean(sentence_lengths) if sentence_lengths else0,
'sentence_length_variance': np.var(sentence_lengths) iflen(sentence_lengths) > 1else0,
'vocab_richness': vocab_richness,
'punctuation_density': punctuation_count / max(len(text), 1)
}
return features
defstyle_distance(style_features_1, style_features_2):
"""
Compute Euclidean distance between style feature vectors.
"""
feat_vec_1 = np.array(list(style_features_1.values()))
feat_vec_2 = np.array(list(style_features_2.values()))
return np.linalg.norm(feat_vec_1 - feat_vec_2)
Step 3: Create Preference Pairs with Deviation Weighting
Convert score-based training data into preference pairs, annotating each with semantic and style deviation values.
defcreate_preference_pairs(training_data):
"""
Transform score-based data (dict with prompt, responses, scores)
into preference pairs with deviation weights. Assumes responses
are sorted by quality score (higher is better).
"""
preference_pairs = []
for example in training_data:
prompt = example['prompt']
responses = example['responses'] # sorted by score
scores = example['scores']
semantic_dev = calculate_deviation(responses)
style_features = [compute_style_features(r) for r in responses]
style_dev = np.mean([
style_distance(style_features[i], style_features[j])
for i inrange(len(style_features))
for j inrange(i + 1, len(style_features))
]) iflen(style_features) > 1else0# Pair highest-scoring response with lower-scoring onesfor i inrange(len(responses) - 1):
win_response = responses[-1] # Highest scored
lose_response = responses[i] # Lower scored# Deviation of winning response
win_dev = (semantic_dev + style_dev) / 2
pair = {
'prompt': prompt,
'winning_response': win_response,
'losing_response': lose_response,
'deviation_weight': win_dev,
'semantic_deviation': semantic_dev,
'style_deviation': style_dev
}
preference_pairs.append(pair)
return preference_pairs
Step 4: Implement Diversified DPO (DDPO)
Extend standard DPO loss by scaling with the deviation weight of the winning response, emphasizing unique high-quality examples.
import torch
import torch.nn.functional as F
defdiversified_dpo_loss(model, batch, beta=0.5):
"""
Diversified DPO loss scales standard DPO by deviation weight.
beta: temperature parameter for preference modeling.
Deviation weight emphasizes rare, high-quality responses.
"""
prompts = batch['prompts']
win_responses = batch['winning_responses']
lose_responses = batch['losing_responses']
deviation_weights = batch['deviation_weights']
# Forward pass for winning responses
win_logits = model.compute_logits(prompts, win_responses)
win_log_probs = F.log_softmax(win_logits, dim=-1).sum(dim=-1)
# Forward pass for losing responses
lose_logits = model.compute_logits(prompts, lose_responses)
lose_log_probs = F.log_softmax(lose_logits, dim=-1).sum(dim=-1)
# Standard DPO loss
log_odds = win_log_probs - lose_log_probs
dpo_loss = -F.logsigmoid(beta * log_odds)
# Scale by deviation weight (higher deviation = higher loss weight)
weighted_loss = dpo_loss * deviation_weights
return weighted_loss.mean()
Step 5: Train with Iterative Refinement
Use the weighted loss in a training loop with a standard optimizer, optionally iterating with data refresh.
deftrain_diversified_model(
model,
preference_pairs,
num_epochs=3,
batch_size=8,
learning_rate=1e-5):
"""
Train model using diversified preference pairs.
Iterates through epochs of preference-based training.
"""
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
for epoch inrange(num_epochs):
total_loss = 0
num_batches = 0# Shuffle and batch preference pairsimport random
random.shuffle(preference_pairs)
for i inrange(0, len(preference_pairs), batch_size):
batch_pairs = preference_pairs[i:i + batch_size]
batch = {
'prompts': [p['prompt'] for p in batch_pairs],
'winning_responses': [p['winning_response'] for p in batch_pairs],
'losing_responses': [p['losing_response'] for p in batch_pairs],
'deviation_weights': torch.tensor(
[p['deviation_weight'] for p in batch_pairs],
dtype=torch.float32
)
}
loss = diversified_dpo_loss(model, batch)
optimizer.zero_grad()
loss.backward()
optimizer.step()
total_loss += loss.item()
num_batches += 1
avg_loss = total_loss / num_batches
print(f"Epoch {epoch + 1} - Avg Loss: {avg_loss:.4f}")
return model
