monotoken

name

monotoken

description

Use when writing or rewriting agent state files, plan documents, phase handoffs, frontier files, or any artifact that will be re-read by an LLM in a long-running autonomous loop (Forge tasks, GSD phases, ralph-loop, subagent handoffs, persistent TODO lists). Compresses prose plans 2-4x via deterministic encoding while preserving comprehension. Do NOT use for user-facing output, security warnings, code, or one-shot prompts.

Monotoken

Token-efficient wire format for LLM agent communication. Saves 30-65% on plan/state files that get re-read every iteration of an autonomous loop.

When to invoke

• Writing PLAN.md inside a GSD phase
• Writing frontier/state files for a Forge task
• Drafting an A2A handoff for a subagent that will read and act on it
• Rewriting a stale planning doc that's bloating context in a long loop
• Persisting TODO lists that an agent re-reads each iteration

When NOT to invoke

• Final output the user will read
• Security warnings, destructive-action confirmations, irreversible ops
• Code blocks, error messages, file paths (always preserved verbatim)
• One-shot prompts where the doc is read once and discarded
• Any context where 2-4x token savings aren't worth a slight comprehension cost

How it works (one paragraph)

Monotoken is not a prompt asking the model to be terse (that's what Caveman does). It is a deterministic encoder that runs at the file I/O boundary. The encoder strips fillers and articles, substitutes multi-word phrases with their single-token equivalents from a tokenizer-derived lexicon, preserves code/paths/errors verbatim, and allows free-English escape via # comment lines. The agent writes English, the hook encodes on save, the agent reads Monotoken on the next iteration. Round-trip comprehension is measured by scripts/measure.js and the encoder falls back to passthrough if a file would lose meaning.

How to use

Manual:

node ~/.claude/skills/monotoken/scripts/encode.js path/to/PLAN.md
node ~/.claude/skills/monotoken/scripts/measure.js path/to/PLAN.md

Automatic (recommended) — add this PostToolUse hook to ~/.claude/settings.json. It only fires on plan/state files, so user experience is unchanged everywhere else:

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Write",
      "hooks": [{
        "type": "command",
        "command": "node C:/Users/20243455/.claude/skills/monotoken/scripts/hook.js"
      }]
    }]
  }
}

The hook script reads the file path from $CLAUDE_TOOL_INPUT, checks whether it matches a planning-file pattern (**/PLAN.md, **/STATE.md, **/frontier.json, **/.planning/**/*.md), and only then encodes in place. All other writes pass through untouched.

Safety rails

1. Allowlist, not denylist. Only encodes files matching planning patterns.
2. Code is sacred. Fenced code blocks, inline code, and file paths pass through verbatim.
3. English escape hatch. Any line starting with # is preserved as-is — use it for nuance.
4. Comprehension gate. measure.js reports the token delta and refuses to encode if the ratio is below 1.3x (not worth it) or above 8x (probably broke something).
5. Reversible. scripts/decode.js reconstructs a readable English version. The encoder is one-way lossy by design (drops fillers), but the structure survives.

Files

• SKILL.md — this file
• lexicon/o200k.txt — single-token English words for o200k_base (Claude/GPT-4o/5)
• lexicon/phrases.json — multi-word -> single-word substitutions
• lexicon/fillers.txt — words/phrases the encoder strips
• scripts/build_lexicon.js — regenerate lexicon from tiktoken
• scripts/encode.js — English -> Monotoken
• scripts/decode.js — Monotoken -> English (sanity check)
• scripts/measure.js — token delta + comprehension safety check
• scripts/hook.js — PostToolUse hook entry point
• examples/ — before/after on real planning docs
• BENCHMARKS.md — measured token reductions

Limitations

• Lexicon is tuned for o200k_base. Other tokenizers (cl100k, Llama, Gemini) get smaller wins because the lexicon entries may split into 2 tokens. Regenerate with build_lexicon.js for your target.
• Caps at ~3-4x compression on prose plans before comprehension degrades.
• Does nothing for code, which is the right behavior — code is already token-dense.
• The "comprehension preserved" claim is measured via token-delta heuristics in v1. A real round-trip LLM evaluation requires API keys and is in BENCHMARKS.md as future work.