llm-diverse-creative-writing

name	llm-diverse-creative-writing
title	Modifying Large Language Model Post-Training for Diverse Creative Writing
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2503.17126
keywords	["LLM","Creative Writing","Diversity","DPO","ORPO","Preference Optimization"]
description	Learn to enhance LLM post-training for diverse creative outputs by weighting training pairs using deviation metrics (semantic and style diversity). Applies to models where standard alignment reduces diversity, enabling competitive quality with higher output variety.

Core Concept

Standard LLM post-training improves output quality but often reduces diversity—a critical limitation for creative writing. This skill introduces Diversified Preference Optimization (DDPO and DORPO), which weights training samples by their deviation from peers sharing the same prompt. Deviation measures how unique and diverse a training example is, prioritizing rare high-quality instances to maintain both quality and diversity.

Architecture Overview

• Deviation Calculation: Computes mean pairwise distance between a training sample and all others with identical prompt using semantic embeddings
• Dual Diversity Metrics: Combines semantic diversity (via Jina embeddings) and style diversity (via specialized embeddings)
• Loss Weighting: Scales DPO/ORPO loss terms by winning response deviation to emphasize diverse examples
• Reward Model: Trained on external signals (e.g., Reddit upvotes) for quality evaluation
• Preference Pair Creation: Transforms score-based training data into preference pairs for DPO/ORPO training

Implementation Steps

Step 1: Calculate Semantic Diversity via Embeddings

This step converts training responses into embeddings and computes deviation as mean pairwise distance within prompt groups.

from sentence_transformers import SentenceTransformer
import numpy as np

# Initialize embedding model
embedding_model = SentenceTransformer('jinaai/jina-embeddings-v3')

def calculate_deviation(responses_for_prompt):
    """
    Calculate deviation (mean pairwise distance) for a list of responses
    sharing the same prompt. Lower values indicate similarity;
    higher values indicate uniqueness.
    """
    embeddings = embedding_model.encode(responses_for_prompt)

    n = len(embeddings)
    if n <= 1:
        return 0.0

    # Compute pairwise distances
    distances = []
    for i in range(n):
         j  (i + , n):
            dist = np.linalg.norm(embeddings[i] - embeddings[j])
            distances.append(dist)

     np.mean(distances)  distances