| name | generated-content-quality |
| description | Keep large bodies of authored-at-volume content from converging: events, quests, items, barks, enemy variants, flavour text, procedural encounters, LLM-written narrative. Use when producing a content pack, when the tenth generated item feels like the third, when per-item review keeps passing everything but the corpus reads samey, when adding to an existing content set, or when deciding what the next batch should contain. Covers the saturation ledger, dimension selection, quota setting, underused-cell seeding, near-duplicate review, and why per-item quality review structurally cannot detect corpus convergence. Applies to human-authored content at volume as much as to generated content. |
Generated Content Quality
Purpose: Stop a content corpus from collapsing into three shapes wearing
forty costumes.
Sources: The saturation-ledger method here is generalised from a working
implementation in the Royal Inbox project (docs/CONTENT_NOVELTY_LEDGER.md),
which measures structural saturation across a 47-event corpus and sets explicit
quotas for the next pack. The underlying convergence problem is documented in
the LLM-diversity literature (mode collapse under RLHF; Kirk et al., "Understanding
the Effects of RLHF on LLM Generalisation and Diversity," 2024). No published
game-industry standard exists for this — the method below is a practice, not a
citation, and its thresholds are posture B under the Numbers Policy.
The structural problem
Per-item quality review cannot detect corpus convergence. This is not a
diligence failure; it is a category error. Each item is reviewed against "is
this good?" — and each item is good. The defect exists only in the
relationship between items, which no per-item pass ever looks at.
The result is a corpus where every entry passes review and the whole reads as
one entry repeated. Players experience it as "samey" and cannot say why, because
the sameness is structural rather than superficial: the surface details all
differ.
Two forces drive it, and both are worse with generated content:
- The attractor. A model asked for a fantasy petition writes a guild, a
council, or a commission — because those are the highest-probability answers,
and they are the highest-probability answer every time you ask.
- Context contamination. Generating item N with items 1..N-1 in context
makes the model more likely to match their shape, not less. "Make it
different from the others" reliably produces a surface variation on the same
structure.
Test: take twenty items from the corpus. Strip the proper nouns and the
surface flavour. Can you tell them apart? If not, the corpus has converged
regardless of how each one scored on its own.
The saturation ledger
The method: pick the dimensions that actually vary, count the corpus along each,
and set quotas for the next batch from the counts.
Step 1 — Choose dimensions
Between five and nine. Each must be something a reader would notice varying, and
each must be classifiable in one or two words per item without a judgement call.
The dimensions are game-specific. Some that work:
| Content type | Dimensions worth counting |
|---|
| Narrative events | Who initiates · what mechanism drives it · setting · the comic or tonal engine · the shape of the choice · what escalates · how it resolves |
| Items / gear | Slot · the fantasy it serves · acquisition path · the trade-off it imposes · visual family · which build it belongs to |
| Enemies | Archetype · what player behaviour it forces · introduction context · counter · silhouette family |
| Quests | Giver class · verb chain · systems touched · reward type · failure state · who is changed by it |
| Barks / flavour | Trigger · register · length · who speaks · whether it references state |
The dimension that matters most is the one you did not think to count. If
every entry in your corpus differs on all five of your dimensions and still reads
samey, you picked the wrong five. Look for the dimension along which everything
is identical — it is usually structural: the shape of the choice, or how things
resolve.
Step 2 — Count what exists
A table, one row per item, one column per dimension. Then the saturation
snapshot, which is the actual output:
35 of 47 senders are named offices, councils, guilds, boards, or commissions.
Governance, bureaucracy and labour account for 24 of 47 themes. Common choice
triangles are accommodate/fund, regulate/tax, and reject/exploit.
That paragraph is worth more than any amount of per-item review, and it takes
twenty minutes to produce.
Step 3 — Set quotas, not aspirations
Convert the counts into hard constraints on the next batch. Quotas, both
directions:
At least eight of sixteen openings come from households, neighbours, farmers,
cooks, healers, teachers, ferrymen, traders, or travelling workers. No more
than four opening senders are named institutions. No more than three are
personified nonhumans. At most one callback may use an audit, hearing, council
or commission as its mechanism.
Ceilings do more work than floors. A floor ("include some domestic senders")
gets satisfied by two tokens. A ceiling ("no more than four institutions") forces
the other twelve somewhere genuinely new, and it is checkable by counting.
Rule: every quota is a number, and someone can count it. "More variety" is
not a quota. It is the wish the quota exists to make real.
Underused-cell seeding
The strongest anti-convergence move available, and it costs nothing:
Seed the next item from a cell of the ledger that is empty or thin.
Not from raw entropy — raw entropy moves you somewhere random, which is often
somewhere already crowded. An underused cell moves you somewhere random and
known-underserved. The ledger has already done the work of identifying where
the corpus is thin; drawing from it converts a diversity problem into a lookup.
Combine with real environment entropy to pick which thin cell:
thin_cells = [c for c in ledger.cells if c.count <= 1]
pick = thin_cells[int(os.urandom(1)[0]) % len(thin_cells)]
The seed is a starting constraint, never the output. A drawn cell still needs an
authored premise, a real choice, and a review pass — it just starts somewhere the
corpus has not been.
Test: after a batch generated this way, recount. If the thin cells are still
thin, the seeding did not survive contact with drafting — which usually means
the generation prompt reasserted the attractor downstream of the seed.
Near-duplicate review
Separate from the ledger and still necessary. The ledger catches structural
convergence; this catches two items that are the same item.
For each new item, ask against the existing corpus:
- Is there an item with the same premise under different flavour?
- Is there an item whose choice set is functionally identical?
- Does it resolve the way something else already resolves?
- Would a player who saw both feel the second one was new?
The last question is the real one, and it is the only one that catches the case
where all the structural dimensions differ and the item is still a repeat.
The referent ledger — the same instrument, run backwards
The saturation ledger catches convergence: too many items sharing a shape.
Its mirror image catches unmanaged divergence: too many words for one thing.
Both are corpus-level defects invisible to per-item review, and the second is
more common in games than the first — because every individual synonym is good
writing. Register rewards variety; comprehension punishes it.
The failure
A logging game's teaching channel refers to the resource the player carries as
wood, timber, tonnage, load, haul, and "the load you
logged". Its delivery point is the truck and the landing. Its shop is an
outpost, a camp, a company outpost, a counter, and a site.
Every one of those is a defensible choice in isolation, and a per-line review
passes all of them. Together they mean a player, seven seconds after the last
notice, resolving a new noun for a thing they already had a name for — while
something is chasing them.
The ledger
One row per game noun. One column: every word the corpus uses for it, and where.
| Referent | Words used | Where | Canon |
|---|
| the resource carried | wood, timber, tonnage, load, haul | notices, shop, ticker | wood |
| the delivery point | truck, landing | notices, shop | truck |
| purchasing power | allowance | shop, notices | allowance |
| a purchasable upgrade | rank, line, shelf item | notices, shop | rank |
| the secondary shop | outpost, camp, counter, site | notices, shop | outpost |
The rule: one canonical word per referent in any channel that teaches. The
synonyms are not deleted — they are reassigned to the flavour channel, where
variety is the point and comprehension is not load-bearing. This is how you keep
the register without paying for it in confusion.
Test: build the table. Any referent with more than one word in a teaching
channel is a bug, and the count is the severity. This takes fifteen minutes on a
corpus of fifty lines and I have never run it without finding something.
Where the line falls
Not every synonym is a defect. The question is whether the reader must resolve
the word to act.
- Flavour, ambient, prose — vary freely. A ticker line calling wood "tonnage"
is characterisation of the company that logs it.
- Anything the player must act on — UI labels, tips, objectives, tutorial
copy, error states — one word, always the same one.
- Proper nouns are exempt but expensive. A named thing can have one name and
one epithet; a third is a second thing.
Where this sits relative to per-item quality
Both are required and they catch different things. Run them in this order:
- Ledger + quotas — before generating. Decides what the batch contains.
Run the referent ledger here too, once, and fix the canon before drafting.
- Underused-cell seeding — during. Decides where each item starts.
- Per-item quality review — after. Voice, correctness, does it work.
- Near-duplicate review — after. Is it actually new.
- Recount — after. Did the saturation snapshot move?
Skipping step 5 is the most common failure. Without it there is no evidence the
process did anything, and the ledger silently becomes documentation of a corpus
nobody is steering.
Applies to human-authored content too
Nothing here is specific to generated text. A single author writing forty events
over three months converges the same way and for a related reason — the attractor
is their own habit rather than a model's prior, and it is equally invisible from
inside. The ledger is the instrument in both cases.
The difference is rate. A human writes forty events in three months and notices
the drift somewhere around item twenty-five. A model writes forty in an afternoon,
and nobody notices until the corpus ships.
Cross-References
Three tight links. Everything else routes through the map, so adding a skill
touches one file rather than twenty: references/routing-map.md (in
game-design).
- game-narrative-craft — Per-item craft for narrative content
- playtest-design — Turning a "samey" report into a countable finding
- data-driven-design — Engagement variance per item, at scale
