| name | token-counter |
| description | In-browser token counter for GPT and other LLM models using tiktoken WASM. Use when you need to count tokens in a text, visualize how text is tokenized, compare token counts of two texts, estimate API costs, or save text snippets for later reference. Triggers include "count tokens", "how many tokens", "token visualization", "API cost estimate", "tiktoken", "compare prompts", "snippet", or any task involving measuring text length for an LLM. |
token-counter
Count tokens in text using the cl100k_base tokenizer (used by GPT-4, GPT-4o, GPT-3.5-turbo). Runs entirely in the browser via tiktoken WASM - no server required.
When to use
- Checking if a prompt fits within a model's context window
- Comparing token counts of two prompt versions
- Estimating API costs before making requests
- Visualizing exactly how a tokenizer splits text
- Saving and comparing snippets across sessions
Running locally
git clone <repo> token-counter
cd token-counter
pnpm install
pnpm dev
Production build:
pnpm build
pnpm preview
No environment variables required. No server. All tokenization runs in the browser.
Pages
| Route | Description |
|---|
| / (Counter) | Main text input and token visualization |
| /compare | Side-by-side comparison of two texts |
| /snippets | Saved snippets list |
| /cost | Cost estimator across all models |
| /guide | Tokenizer reference guide |
Supported models
| Model | Tokenizer | Input per 1M | Output per 1M |
|---|
| gpt-4o | cl100k_base (exact) | $5.00 | $15.00 |
| gpt-4 | cl100k_base (exact) | $30.00 | $60.00 |
| gpt-4-turbo | cl100k_base (exact) | $10.00 | $30.00 |
| gpt-3.5-turbo | cl100k_base (exact) | $0.50 | $1.50 |
| claude-3-5-sonnet | ~4 chars/token | $3.00 | $15.00 |
| claude-3-opus | ~4 chars/token | $15.00 | $75.00 |
| gemini-1.5-pro | ~4 chars/token | $1.25 | $5.00 |
| llama-3-70b | ~4 chars/token | $0.59 | $0.79 |
Models marked "~4 chars/token" use a character-based approximation since they do not use cl100k_base. The UI labels these as "(approximate)".
Token visualization
Each token in the visualization panel is rendered as a colored <span> with a cycled 20-color palette. When "Show token IDs" is enabled, the integer vocabulary index is displayed below each span.
Example:
- "tokenizer" splits into:
token (ID: 4037) + izer (ID: 7609) = 2 tokens
- "Hello" = 1 token (ID: 15339)
- " world" (with leading space) = 1 token (ID: 1917)
Keyboard shortcuts
| Shortcut | Action |
|---|
| Ctrl+Enter | Save current text as snippet |
| Ctrl+K | Clear the text input |
localStorage
| Key | Description |
|---|
| token-counter-snippets | JSON array of saved Snippet objects |
| token-counter-settings | Settings object (default model, output ratio, etc.) |
No data is sent to any server. All storage is local to the browser.
Cost estimation
The Cost page allows configuring:
- Input token count (manual entry or pulled from Counter)
- Output ratio (output = input x ratio, default 0.5x)
- Requests per day (for monthly projection)
All 8 models are compared in a sortable table. The cheapest model for the given token count is highlighted in green.
Tips for reducing token usage
- Remove extra whitespace and blank lines
- Use compact JSON (no indentation)
- Keep system prompts under 200 tokens
- Use common, short words instead of rare or long ones
- Truncate retrieved context chunks in RAG pipelines
See the Guide page (/guide) for a full explanation of how cl100k_base works and common token counts by content type.
