token-optimizer

name	token-optimizer
description	Reduces LLM token consumption by 60-90% on common commands. Filters and compresses command outputs before they reach LLM context. Use when executing shell commands, reading files, running tests, or any operation that produces verbose output. Inspired by RTK (Rust Token Killer) architecture.

Token Optimizer Skill

Minimize LLM token consumption through intelligent output filtering and compression

This skill applies RTK-style filtering strategies to reduce token usage by 60-90% while preserving essential information.

Core Principles

1. Filter before sending - Compress output before it reaches the LLM
2. Preserve signal, remove noise - Keep errors, failures, and changes; hide passing tests, progress bars, and boilerplate
3. Progressive disclosure - Show summary by default, provide full output on demand
4. Track savings - Record token savings for analytics

Filtering Strategies

1. Stats Extraction (90-99% reduction)

For commands that produce verbose success output:

# Instead of full git status output (200 tokens)
# Send: "3 modified, 1 untracked ✓" (~10 tokens)

# Instead of full test output (5000 tokens)
# Send: "✓ 18/20 tests passed, 2 failed" (~8 tokens)

Apply to: git status, git log, git diff, test runners, build tools

2. Error-Only Mode (60-80% reduction)

For build/test/lint commands:

# Show only failures and warnings
# Hide: passing tests, successful builds, info messages

# Example output:
❌ test_auth_login: AssertionError at line 42
⚠️ 3 deprecation warnings
✓ 47/50 tests passed

Apply to: cargo test, npm test, pytest, tsc, eslint

3. Grouping & Aggregation (80-90% reduction)

Group similar items:

# Instead of listing 100 lint errors individually
# Send:
no-unused-vars: 23 occurrences
  - src/utils.ts:5,12
  - src/api.ts:8
semi: 45 occurrences
  (truncated, see full log)

Apply to: linters, type checkers, search results

4. Tree Compression (50-70% reduction)

Compress directory listings:

# Instead of flat ls -la (800 tokens)
# Send:
src/ (8 files, 12KB)
  ├─ main.rs
  ├─ utils.rs
  └─ lib/ (12 files)
tests/ (5 files)
Cargo.toml
README.md

Apply to: ls, tree, find

5. Code Filtering (60-90% reduction)

Strip non-essential code:

// FilterLevel::Minimal - Strip comments only
fn calculate_total(items: &[Item]) -> i32 {
    items.iter().map(|i| i.value).sum()
}

// FilterLevel::Aggressive - Strip function bodies
fn calculate_total(items: &[Item]) -> i32 { ... }

Apply to: cat, read, code review

6. Progress Filtering (85-95% reduction)

Strip progress bars and live updates:

# Instead of animated progress:
# [====================> ] 85%  00:02:15
# Sending: "✓ Downloaded 125MB"

Apply to: wget, curl, npm install, docker pull

7. Deduplication (70-85% reduction)

Collapse repeated lines:

# Instead of 50 identical error lines:
# Send: "[ERROR] Connection timeout (×50)"

Apply to: logs, repeated errors

8. Structure-Only for JSON (80-95% reduction)

Show JSON schema without values:

// Instead of full JSON (1000 tokens)
// Send:
{
  "user": { "id": "...", "name": "...", "email": "..." },
  "posts": [ { "title": "...", "date": "..." }, ... ]
}

Apply to: JSON responses, config files

Command-Specific Filters

Git Operations

Command	Filter Strategy	Example Output
git status	Stats extraction	"3 modified, 1 untracked ✓"
git log -n 5	One-line summary	"5 commits, +142/-89"
git diff	Condensed diff	"3 files: +25, -12"
git push	Result only	"✓ main → origin"
git add	Confirmation	"✓ 3 files staged"

Test Runners

Command	Filter Strategy	Example Output
cargo test	Failures only	"❌ 2/50 failed: test_auth, test_parse"
pytest	Failures + summary	"❌ 2 failed, ✓ 48 passed"
vitest	Compact mode	"2 fail (see details below)"
go test	NDJSON summary	"pkg1 ✓, pkg2 ❌ (3 fails)"

File Operations

Command	Filter Strategy	Example Output
ls -la	Tree compression	"src/ (8), tests/ (5), +3 files"
cat file.rs	Code filtering	"fn main() { ... } (45 lines)"
find . -name "*.rs"	Grouped	"src/: 12, tests/: 5"
grep "pattern"	Grouped by file	"main.rs: 5, utils.rs: 3"

Build Tools

Command	Filter Strategy	Example Output
cargo build	Errors only	"✓ Built in 3.2s" or "❌ error[E0425]"
npm run build	Summary	"✓ Built (2 warnings)"
tsc	Grouped errors	"no-unused-vars: 12, semi: 5"
next build	Final status	"✓ Compiled successfully"

Usage Patterns

Basic Usage

# Standard (auto-detects and applies optimal filter)
rtk git status
rtk cargo test
rtk ls -la

# Verbose (show full output)
rtk git status -v
rtk cargo test -vv

# Ultra-compact (maximum compression)
rtk git status -u
rtk ls -u

In LLM Context

When sending command output to LLM:

## Command Output (Filtered, 90% token savings)

✓ 18/20 tests passed ❌ 2 failed:

• test_auth_login: AssertionError at line 42
• test_parse: panic at utils.rs:18

[Full log: ~/.local/share/rtk/tee/1707753600.log]

Tee Mechanism (Full Output Recovery)

When a command fails or user requests verbose mode, save full output:

# Automatically saved on failure
~/.local/share/rtk/tee/<timestamp>_<command>.log

# LLM can reference without re-executing
[Full output: ~/.local/share/rtk/tee/1707753600_cargo_test.log]

Token Tracking

Track token savings over time:

# Summary statistics
rtk gain

# Output example:
Total commands: 1,247
Original tokens: 2.4M
Filtered tokens: 480K
Saved: 1.9M tokens (80%)
Avg savings per command: 1,560 tokens

# Graph (last 30 days)
rtk gain --graph

# Daily breakdown
rtk gain --daily

Implementation Guide

Creating a Filter Module

// Example: git_status_filter.rs
pub fn filter_git_status(output: &str) -> String {
    let mut modified = 0;
    let mut untracked = 0;
    let mut staged = 0;
    
    for line in output.lines() {
        if line.starts_with("M") { modified += 1; }
        if line.starts_with("??") { untracked += 1; }
        if line.starts_with("A") { staged += 1; }
    }
    
    format!(
        "{} modified, {} untracked, {} staged ✓",
        modified, untracked, staged
    )
}

Bash Script Version

#!/bin/bash
# scripts/filter-git-status.sh

output=$(git status --porcelain)
modified=$(echo "$output" | grep -c "^ M")
untracked=$(echo "$output" | grep -c "^??")
staged=$(echo "$output" | grep -c "^A")

echo "$modified modified, $untracked untracked, $staged staged ✓"

Best Practices

DO ✅

• Show summary statistics (counts, percentages)
• Highlight failures and errors
• Provide file paths for issues
• Include exit codes
• Offer full output path on demand
• Use color/icons for quick scanning

DON'T ❌

• Send full success logs
• Include progress bars
• Repeat identical lines
• Show boilerplate headers/footers
• Send ANSI escape codes unnecessarily
• Hide critical errors in verbose output

Verbosity Levels

Level	Flag	Behavior
Normal	(none)	Filtered summary
Debug	-v	Show debug messages
Verbose	-vv	Show command being executed
Raw	-vvv	Show full unfiltered output

Configuration

Optional config file: ~/.config/token-optimizer/config.toml

[tracking]
enabled = true
database_path = "~/.local/share/token-optimizer/history.db"

[filters]
default_level = "smart"  # smart, minimal, aggressive
save_tee_on_failure = true
max_tee_files = 20

[commands]
# Override default filters for specific commands
git_status = "stats"
cargo_test = "failures-only"
ls = "tree"

Examples

Example 1: Test Failure Analysis

Without optimization (5000 tokens):

running 50 tests
test test_auth_login ... ok
test test_auth_logout ... ok
... (47 more lines)
test test_parse ... FAILED
thread 'test_parse' panicked at 'assertion failed', src/utils.rs:18:5
...

With optimization (50 tokens):

❌ 2/50 tests failed

test_parse: panic at utils.rs:18
test_auth_login: AssertionError at line 42

[Full log: ~/.local/share/rtk/tee/1707753600.log]

Example 2: Directory Listing

Without optimization (800 tokens):

drwxr-xr-x 15 user staff 480 Jan 1 12:00 .
drwxr-xr-x 12 user staff 384 Jan 1 12:00 ..
-rw-r--r--  1 user staff  12K Jan 1 12:00 main.rs
... (45 more lines)

With optimization (100 tokens):

src/ (8 files, 45KB)
  ├─ main.rs (12K)
  ├─ utils.rs (8K)
  └─ lib/ (12 files)
tests/ (5 files)
Cargo.toml
README.md

Example 3: Git Diff

Without optimization (2000 tokens):

diff --git a/src/main.rs b/src/main.rs
index abc123..def456 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -15,7 +15,7 @@ fn main() {
-    println!("Hello");
+    println!("Hello, World!");
... (200 more lines)

With optimization (80 tokens):

3 files changed
  src/main.rs: +5, -2
  src/utils.rs: +12, -8
  tests/test_auth.rs: +8, -2

Total: +25, -12

Integration with LLM Agents

For Claude Code

Add to ~/.claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "command": "token-opt-rewrite.sh"
      }
    ]
  }
}

For OpenCode

Create ~/.config/opencode/plugins/token-optimizer.ts:

export default {
  hooks: {
    'tool.execute.before': async (tool) => {
      if (tool.type === 'bash') {
        return { ...tool, command: `rtk ${tool.command}` };
      }
      return tool;
    }
  }
};

Metrics & Analytics

Track these metrics:

1. Token Savings: Original vs filtered tokens
2. Command Coverage: % of commands optimized
3. User Satisfaction: Did filter preserve needed info?
4. Performance: Filter overhead (<10ms target)

Troubleshooting

Filter removed important info

Solution: Use -v flag or check tee file

False positive on error detection

Solution: Adjust error patterns in filter module

Performance overhead too high

Solution: Simplify filter logic, use early returns

References

• RTK Architecture
• RTK Filtering Strategies
• Token Economics

License

MIT - Based on RTK architecture