chain-of-density

name	chain-of-density
description	Iteratively densify text summaries using Chain-of-Density technique. Use when compressing verbose documentation, condensing requirements, or creating executive summaries while preserving information density.
license	Apache-2.0
compatibility	Python 3.10+ (for text_metrics.py script via uv run)
metadata	{"author":"agentic-insights","version":"1.2","paper":"From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting","arxiv":"https://arxiv.org/abs/2309.04269"}

Chain-of-Density Summarization

Compress text through iterative entity injection following the CoD paper methodology. Each pass identifies missing entities from the source and incorporates them while maintaining identical length.

The Method

Chain-of-Density works through multiple iterations:

1. Iteration 1: Create sparse, verbose base summary (4-5 sentences at target_words)
2. Subsequent iterations: Each iteration:
   • Identify 1-3 missing entities from SOURCE (not summary)
   • Rewrite summary to include them
   • Maintain IDENTICAL word count through compression

Key principle: Never drop entities - only add and compress.

Missing Entity Criteria

Each entity added must meet ALL 5 criteria:

Criterion	Description
Relevant	To the main story/topic
Specific	Descriptive yet concise (≤5 words)
Novel	Not in the previous summary
Faithful	Present in the source (no hallucination)
Anywhere	Can be from anywhere in the source

Quick Start

1. User provides text to summarize
2. Orchestrate 5 iterations via cod-iteration agent
3. Each iteration reports entities added via Missing_Entities: line
4. Return final summary + entity accumulation history

Orchestration Pattern

Iteration 1: Sparse base (target_words, verbose filler)
     ↓ Missing_Entities: (none - establishing base)
Iteration 2: +3 entities, compress filler
     ↓ Missing_Entities: "entity1"; "entity2"; "entity3"
Iteration 3: +3 entities, compress more
     ↓ Missing_Entities: "entity4"; "entity5"; "entity6"
Iteration 4: +2 entities, tighten
     ↓ Missing_Entities: "entity7"; "entity8"
Iteration 5: +1-2 entities, final density
     ↓ Missing_Entities: "entity9"
Final dense summary (same word count, 9+ entities)

How to Orchestrate

Iteration 1 - Pass source text only:

Task(subagent_type="cod-iteration", prompt="""
iteration: 1
target_words: 80
text: [SOURCE TEXT HERE]
""")

Iterations 2-5 - Pass BOTH previous summary AND source:

Task(subagent_type="cod-iteration", prompt="""
iteration: 2
target_words: 80
text: [PREVIOUS SUMMARY HERE]
source: [ORIGINAL SOURCE TEXT HERE]
""")

Critical:

• Invoke serially, not parallel
• Pass SOURCE text in every iteration for entity discovery
• Parse Missing_Entities: line to track entity accumulation

Expected Agent Output Format

The cod-iteration agent returns:

Missing_Entities: "entity1"; "entity2"; "entity3"

Denser_Summary:
[The densified summary - identical word count to previous]

Parse both parts - track entities for history, pass summary to next iteration.

Measuring Density

Use scripts/text_metrics.py for deterministic word counts:

echo "your summary text" | uv run scripts/text_metrics.py words
# Returns: word count

uv run scripts/text_metrics.py metrics "your summary text"
# Returns: {"words": N, "chars": N, "bytes": N}

Parameters

Parameter	Default	Description
iterations	5	Number of density passes (paper uses 5)
target_words	80	Word count maintained across ALL iterations
return_history	false	Include intermediate summaries + entities

Note: target_words can be adjusted based on source length and desired output density.

Output Format

Minimal (default)

[Final dense summary text]

With History (return_history=true)

final_summary: |
  [Dense summary at target_words with accumulated entities]
iterations:
  - turn: 1
    missing_entities: "(none - establishing base)"
    words: 80
    summary: |
      [Sparse iteration 1]
  - turn: 2
    missing_entities: "entity1; entity2; entity3"
    words: 80
    summary: |
      [Denser iteration 2]
  # ... etc
total_entities: 9

When to Use

• Verbose documentation exceeding 500 words
• Requirements documents needing condensation
• Creating executive summaries from detailed reports
• Compressing skills that exceed recommended length

When NOT to Use

• Legal/compliance text (precision required)
• Tutorial content (beginners need explanation)
• Already concise content (<300 words)
• Specifications (don't compress specs)

Example

Source (180 words, verbose skill excerpt):

The name field is a required field that must be present in every skill.
The name field identifies the skill and must follow a specific format.
For the name field, you should use lowercase letters and hyphens only.
The name field can be 1 to 64 characters long. The description field
is also required and tells agents when to use your skill...

Iteration 1 (Sparse, 80 words):

Missing_Entities: (none - establishing base)

Denser_Summary:
This document discusses the requirements for skill configuration fields in agent systems. It covers various aspects of how fields should be formatted and what values they can contain. The document also mentions validation rules that apply to these fields and provides guidance on best practices for developers. Additionally, it includes information about both optional and required elements that need to be considered when creating new skills for the system.

Iteration 3 (After 2 passes, same 80 words):

Missing_Entities: "1-64 characters"; "lowercase alphanumeric-hyphens"; "Use when phrase"

Denser_Summary:
Skills require `name` (1-64 chars, lowercase alphanumeric-hyphens) and `description` fields with validation rules. The name identifies skills; descriptions tell agents when to invoke using "Use when..." phrases. Both fields have format constraints and best practices. Optional metadata fields provide author, version, and compatibility information for cross-platform agent discovery.

Final Iteration 5 (Same 80 words, maximum density):

Missing_Entities: "Claude Code"; "Cursor"; "GitHub Copilot"

Denser_Summary:
Required: `name` (1-64 chars, ^[a-z0-9]+(-[a-z0-9]+)*$) and `description` (1-1024 chars) with validation. Description includes "Use when..." + discovery keywords for auto-invocation. Optional: license (SPDX), compatibility, metadata (author, version, tags). Cross-platform: Claude Code, Cursor, GitHub Copilot. Name matches directory. Progressive disclosure via references/, assets/, scripts/ subdirectories.

Architecture Note

This skill implements the CoD paper methodology:

• Skill = orchestrator (this file)
• Agent = stateless worker (cod-iteration)
• Script = deterministic utility (text_metrics.py)

Sub-agents cannot call other sub-agents. Only skills orchestrate via Task tool.

References

• Paper: From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting
• Dataset: HuggingFace griffin/chain_of_density