convex-tokenization

name	convex-tokenization
description	ConvexTok methodology — formulating tokenizer construction as a convex optimization (linear program) instead of greedy BPE/Unigram. Use when: (1) Designing tokenizers for new languages or domains, (2) Improving bits-per-byte (BpB) efficiency of LLM tokenizers, (3) Evaluating tokenizer quality beyond greedy heuristics, (4) Tokenizer research comparing BPE/Unigram vs. globally optimal approaches.
arxiv_id	2605.22821
date	2026-05-21
authors	Unknown
tags	["information-science","tokenization","nlp","convex-optimization","llm-preprocessing"]

ConvexTok: Tokenization via Convex Relaxations

Description

Current tokenizer algorithms (BPE, Unigram) are greedy — they make locally optimal decisions without considering the vocabulary as a whole. ConvexTok formulates tokenizer construction as a linear program and solves it using convex optimization tools, yielding globally better tokenizers.

Core Insight

Problem: BPE/Unigram make greedy merge/split decisions → suboptimal vocabulary.

Solution: Formulate as LP with constraints on vocabulary size, coverage, and compression → solve globally.

Mathematical Framework

The tokenization problem is formulated as:

min Σ_i c_i · x_i
subject to:
  Σ_i A_ij · x_i ≥ 1  (coverage constraint)
  Σ_i x_i ≤ V          (vocabulary size constraint)
  x_i ∈ {0, 1}         (selection variable)

Where:

• x_i = whether subword i is in vocabulary
• c_i = cost (frequency-weighted length) of subword i
• A_ij = coverage matrix (subword i covers position j)
• V = target vocabulary size

Relax to x_i ∈ [0, 1] for convex solution, then round.

Key Advantages

1. Global optimization: Considers full vocabulary, not greedy merges
2. Intrinsic metrics: Consistently improves BpB (bits-per-byte)
3. Flexible constraints: Can add domain-specific constraints
4. Theoretical guarantees: Convex relaxation provides optimality bounds

Usage Patterns

Pattern 1: Tokenizer Design for New Domain

When building a tokenizer for a specialized domain (medical, legal, code):

1. Collect domain corpus
2. Build candidate subword inventory (all n-grams up to max length)
3. Formulate LP with domain-specific costs
4. Solve with convex optimizer (CVX, scipy.optimize)
5. Round solution to integer vocabulary
6. Validate BpB on held-out data

Pattern 2: Tokenizer Evaluation

Compare existing tokenizers against ConvexTok upper bound:

1. Run ConvexTok on same corpus as BPE/Unigram
2. Compare BpB: ConvexTok provides theoretical upper bound
3. Gap analysis: large gap → greedy tokenizer is far from optimal

Pattern 3: Hybrid Approach

Use ConvexTok to initialize vocabulary, then fine-tune:

1. Run ConvexTok to get globally optimal initial vocabulary
2. Use as starting point for further BPE merges
3. Achieve better final tokenizer than pure BPE

Implementation Guidance

LP Solver Selection

• Small vocabularies (<100K candidates): scipy.optimize.linprog
• Medium (100K-1M): cvxpy with ECOS or SCS solver
• Large (>1M): Column generation + specialized solver

Rounding Strategy

After solving LP relaxation:

1. Threshold rounding: x_i > 0.5 → include
2. Greedy rounding: Select highest fractional values until budget
3. Pipage rounding: Preserves LP objective bounds

Error Handling

LP Infeasible

• Check vocabulary size constraint V is feasible
• Verify coverage constraints aren't contradictory
• Relax constraints iteratively

Slow Solve Time

• Use column generation: start with small candidate set
• Add promising subwords iteratively
• Warm-start from BPE vocabulary

Resources

• arXiv: 2605.22821
• Related: BPE, Unigram, SentencePiece, convex optimization