superwriter-longform

name	superwriter-longform
title	SuperWriter: Reflection-Driven Long-Form Generation with LLMs
version	0.0.2
engine	skillxiv-v0.0.2-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2506.04180
keywords	["long-form-generation","reflection","tree-search","direct-preference-optimization"]
description	Generate coherent, consistent long-form text through structured planning, hierarchical reflection, and Monte Carlo tree search-guided optimization.

SuperWriter: Reflection-Driven Long-Form Generation

Core Concept

SuperWriter-Agent demonstrates that long-form text generation quality improves through explicit structured thinking stages that mimic professional writing workflows. By combining hierarchical direct preference optimization with Monte Carlo tree search (MCTS) for propagating quality signals, a 7B SuperWriter model outperforms larger baselines across diverse long-form writing tasks.

Architecture Overview

• Structured Thinking Stages: Decompose generation into planning, drafting, and refinement phases
• Hierarchical Direct Preference Optimization (DPO): Use MCTS to propagate quality judgments from document level down to paragraph and sentence levels
• Reflection Mechanism: Generate internal critiques and improvement suggestions to guide refinement
• SuperWriter-LM: Fine-tuned 7B model trained on structured reasoning dataset
• Goal: Maintain coherence and logical consistency across extended sequences

Implementation

Step 1: Prepare Structured Thinking Dataset

from typing import List, Dict
import json

class LongFormDatasetBuilder:
    def __init__(self):
        self.structured_templates = {
            'article': self.article_structure(),
            'report': self.report_structure(),
            'narrative': self.narrative_structure(),
        }

    def article_structure(self):
        """Extract structure from well-written articles"""
        return {
            'outline': 'Hierarchical topic breakdown',
            'introduction': 'Hook + thesis statement',
            'body_paragraphs': 'Topic sentence + evidence + analysis',
            'conclusion': 'Summary + forward thinking',
        }

    def build_structured_examples(self, raw_documents: List[str]):
        """Decompose long documents into thinking stages"""
        structured_data = []

        for doc in raw_documents:
            # Stage 1: Extract or generate outline
            outline = self.extract_outline(doc)

            # Stage 2: Identify structural elements
            structure = self.identify_structure(doc)

            # Stage 3: Create supervision signal
            example = {
                : doc,
                : outline,
                : structure,
                : .generate_thinking_chain(doc),
                : .assess_coherence(doc),
            }

            structured_data.append(example)

         structured_data

     () -> :
        
         {
            : {
                : .extract_main_topic(document),
                : .extract_key_points(document),
                : .infer_audience(document),
                : .estimate_scope(document),
            },
            : {
                : .extract_claims(document),
                : .extract_evidence(document),
                : .identify_transitions(document),
            },
            : {
                : .suggest_clarity_fixes(document),
                : .find_inconsistencies(document),
                : .suggest_style_improvements(document),
            }
        }


builder = LongFormDatasetBuilder()
structured_dataset = builder.build_structured_examples(documents)

Step 2: Implement Multi-Stage Generation

class SuperWriterAgent:
    def __init__(self, base_model_name='Qwen-7B'):
        self.model = load_model(base_model_name)
        self.planner = self.PlanningModule()
        self.drafter = self.DraftingModule()
        self.reflector = self.ReflectionModule()

    class PlanningModule:
        def generate_outline(self, topic: str, target_length: int) -> Dict:
            """Stage 1: Generate structured outline"""
            prompt = f"""Given topic: {topic}
Target length: {target_length} words

Generate a detailed outline with:
1. Main thesis
2. Key sections (3-5)
3. Key points per section (2-3 each)
4. Logical flow between sections

Format as nested structure."""

            outline = generate_completion(prompt)
            return self.parse_outline(outline)

        def parse_outline(self, outline_text: str) -> Dict:
            """Structure outline into actionable sections"""
            sections = []
            current_section = None

            for line in outline_text.split('\n'):
                if line.startswith('1.'):
                    current_section = {'title': line[2:], : []}
                 line.startswith():
                    current_section[].append(line[:])
                 current_section:
                    sections.append(current_section)

             {: sections}

     :
         () -> :
            
            draft_text = 

            
            intro_prompt = 
            introduction = generate_completion(intro_prompt)
            draft_text += introduction + 

            
             section  outline[]:
                section_prompt = 

                section_text = generate_completion(section_prompt)
                draft_text += section_text + 

            
            conclusion_prompt = 
            conclusion = generate_completion(conclusion_prompt)
            draft_text += conclusion

             draft_text

     :
         () -> :
            
            critique_prompt = 

            critique = generate_completion(critique_prompt)

             {
                : critique,
                : .score_issues(critique),
                : .prioritize_fixes(critique),
            }

         () -> :
            
            issues = {}

             issue_type  [, , ]:
                score_prompt = 
                score = extract_numeric_score(generate_completion(score_prompt))
                issues[issue_type] = score

             issues

         () -> :
            
            refinement_prompt = 

            refined = generate_completion(refinement_prompt)
             refined

     () -> :
        

        
        outline = .planner.generate_outline(topic, target_length)

        
        draft = .drafter.draft_from_outline(outline, topic)

        
        critique = .reflector.generate_critique(draft)
        refined = .reflector.suggest_refinements(draft, critique)

         {
            : outline,
            : draft,
            : critique,
            : refined,
            : refined,  
        }

Step 3: Implement Hierarchical DPO with MCTS

import numpy as np
from collections import defaultdict

class HierarchicalDPOWithMCTS:
    def __init__(self, base_model, value_model):
        self.base_model = base_model
        self.value_model = value_model  # Evaluates quality at different levels

    def evaluate_document_quality(self, document: str) -> float:
        """Document-level quality assessment"""
        metrics = {
            'coherence': self.assess_coherence(document),
            'consistency': self.assess_consistency(document),
            'completeness': self.assess_completeness(document),
            'clarity': self.assess_clarity(document),
        }

        return np.mean(list(metrics.values()))

    def hierarchical_evaluation(self, document: str) -> Dict:
        """Multi-level quality scoring"""
        paragraphs = document.split('\n\n')
        sentences = [s for para in paragraphs for s in para.split('. ')]

        return {
            'document_score': self.evaluate_document_quality(document),
            : [.assess_paragraph(p)  p  paragraphs],
            : [.assess_sentence(s)  s  sentences],
        }

     ():
        

        better_doc = .select_better_document(document_pair)
        worse_doc = document_pair[]  document_pair[] == better_doc  document_pair[]

        better_paras = better_doc.split()
        worse_paras = worse_doc.split()

        paragraph_preferences = []

         b_para, w_para  (better_paras, worse_paras):
            
            visit_count = defaultdict()
            value_sum = defaultdict()

             iteration  (max_iterations):
                
                simulated = .simulate_paragraph_edit(w_para, b_para)

                
                quality = .assess_paragraph(simulated)

                
                edit_signature = (simulated)
                visit_count[edit_signature] += 
                value_sum[edit_signature] += quality

            
            best_edit = (visit_count.keys(),
                          key= x: value_sum[x] / visit_count[x])

            paragraph_preferences.append({
                : b_para,
                : w_para,
                : best_edit,
                : value_sum[best_edit] / visit_count[best_edit]
            })

         paragraph_preferences

     ():
        

        
        prefs = .hierarchical_evaluation(document_pair[])
        worse_prefs = .hierarchical_evaluation(document_pair[])

        
        para_preferences = .mcts_preference_propagation(document_pair)

        
        
        log_prob_better = .base_model.log_probability(document_pair[])
        log_prob_worse = .base_model.log_probability(document_pair[])

        
        dpo_loss = -np.log(
            torch.sigmoid(log_prob_better - log_prob_worse)
        )

        optimizer.zero_grad()
        dpo_loss.backward()
        optimizer.step()

         dpo_loss.item()

Practical Guidance

1. Multi-Stage Decomposition: Always split long-form generation into explicit stages: planning (outline), drafting (section by section), reflection (critique), refinement (editing based on critique).

2. Enforce Outline Adherence: Have the drafter explicitly reference outline sections to maintain logical flow. Check that each paragraph maps to outline points.

3. Implement Reflection Loop: Generate internal critiques before refinement. This "thinking aloud" about weaknesses significantly improves final output quality.

4. MCTS for Quality Propagation: Use Monte Carlo tree search to understand which paragraph-level structures align with document-level quality. This avoids needing paragraph-level human annotations.

5. Hierarchical Optimization: Train with preferences at multiple levels—document, paragraph, sentence—rather than just document-level scores. This helps internalize writing patterns at all scales.

6. Target Model Size: A 7B model with structured training outperforms much larger generalist models on long-form tasks. Quality of training process matters more than parameter count.

Reference

• Paper: SuperWriter (2506.04180)
• Architecture: 7B SuperWriter-LM with planning, drafting, reflection modules
• Method: Hierarchical DPO with MCTS preference propagation
• Key Insight: Explicit structured thinking improves long-form generation coherence