token-reduction-ultimate

name

token-reduction-ultimate

description

The ultimate global skill to maximize token savings using the full SuperCLI ecosystem. Combines rtk, tokf, context-mode, token-optimizer-cli, codefetch, project-map, yek, probe, colgrep, codedb, sloc-guard, and chainlink into a unified decision-driven workflow. Covers discovery via `sc bin`, installation, smoke-testing, and daily usage patterns proven to save 60-98% on token consumption. Use when the user says "smarter", "budget", "rtk", "token", "compress", "optimize context", "reduce tokens", "make it cheaper", "context window", "bare metal", "clean machine", "nothing installed", "no tools", or any mention of AI context management or installing from scratch.

Token Reduction Ultimate

Triggers: smarter · budget · rtk · slowql · token · compress · optimize · reduce tokens · make it cheaper · context window · save tokens · context optimization · bloat · LLM costs · git json · SQL analyze · bare metal · clean machine · nothing installed · no tools · bootstrap

Maximize your AI context budget by chaining smart proxies, sandboxed execution, semantic search, compressed serialization, and code intelligence — all discovered and orchestrated through SuperCLI (sc bin / sc discover).

Architecture

                    ┌──────────────────────────────┐
                    │   AI Agent (Claude, etc.)    │
                    └──────────┬───────────────────┘
                               │
          ┌────────────────────┼────────────────────┐
          │                    │                    │
          ▼                    ▼                    ▼
   ┌────────────┐     ┌──────────────┐     ┌──────────────┐
   │  rtk /     │     │ context-mode │     │  tokf        │
   │  Command   │     │  Sandboxed   │     │  Config      │
   │  Proxy     │     │  Subprocess  │     │  Compressor  │
   │  60-90%    │     │  up to 98%   │     │  Custom      │
   └────────────┘     └──────────────┘     └──────────────┘
          │                    │                    │
          └────────────────────┼────────────────────┘
                               │
                    ┌──────────▼──────────┐
                    │   Search / Analyze  │
                    │  probe  colgrep     │
                    │  codedb  project-map│
                    │  yek  codefetch     │
                    └─────────────────────┘

Bare Metal — No Tools Installed?

If this machine has none of these tools (not even sc), don't panic. This section tells you exactly what to do depending on what IS available.

Step 0: Inventory what you already have

# Check what package managers and runtimes exist
which curl wget npm cargo go rustc pip python3 brew apt pacman 2>/dev/null | head -20

Minimal bootstrap — install sc first

The sc binary comes from the superacli npm package. Three proven methods:

# Option A: via npm (if Node.js is available — most common)
# The npm package is 'superacli' — it installs the 'sc' binary
npm install -g superacli
# Verify
sc --version

# Option B: Zig binary (single static binary, no deps, fastest)
# Download from GitHub releases — picks the right platform automatically
curl -sSL https://github.com/javimosch/supercli/releases/download/v0.1.0-zig/install.sh | bash -s -- --path ~/.local/bin
# Or download Linux amd64 directly
curl -sL https://github.com/javimosch/supercli/releases/download/v0.1.0-zig/sc-zig-linux-amd64 -o ~/.local/bin/sc-zig && chmod +x ~/.local/bin/sc-zig
# Verify
sc-zig --version

# Option C: raw download (if curl/wget exist but no npm)
# Pick the right URL for your platform:
#   Linux amd64:  sc-zig-linux-amd64
#   Linux arm64:  sc-zig-linux-arm64
#   macOS amd64:  sc-zig-darwin-amd64
#   macOS arm64:  sc-zig-darwin-arm64
curl -sL https://github.com/javimosch/supercli/releases/download/v0.1.0-zig/sc-zig-linux-amd64 -o ~/.local/bin/sc && chmod +x ~/.local/bin/sc
# Verify
sc --version

Note: The npm package is superacli, not supercli. The binary is sc. Once installed, sc plugins install <name> discovers and installs all other token-reduction tools automatically.

Then use sc to install everything else

# sc automatically discovers and installs these token-reduction plugins
# via the plugin registry (no manual downloads needed)

# Tier 1 — primary token reduction
sc plugins install rtk             # Rust Token Killer
sc plugins install tokf            # Token Filter
sc plugins install context-mode    # MCP sandbox

# Tier 2 — codebase optimization
sc plugins install token-optimizer-cli
sc plugins install codefetch
sc plugins install project-map
sc plugins install yek

# Tier 3 — search efficiency
sc plugins install probe
sc plugins install colgrep
sc plugins install codedb

# Tier 4-5 — quality & sessions
sc plugins install slowql          # SQL static analyzer
sc plugins install sloc-guard
sc plugins install chainlink

What if there's literally no package manager at all?

If the machine has nothing (no curl, wget, npm, cargo, go, rustc, pip, python, brew, apt):

Built-in tool	Token-saving equivalent	How
ls	rtk ls	Use ls -m for comma-separated, pipe to head
tree -d -L 2	project-map	find . -type d -maxdepth 2 | sort
wc -l	token-count	wc -l -w -c for counts
grep -rn	probe / colgrep	grep -rnl "pattern" --include="*.ts"
git diff --stat	rtk git diff	git diff --stat is already compact
du -sh *	tokf run du	du -sh * | sort -rh
head -n 50	context-mode	Limit reads with head before dumping to agent
cat | nl	yek	cat file.ts | nl for line-numbered output

Core principle for bare-metal: Never dump raw output into agent context. Always grep, head, wc, sort -u first. Pre-process on the machine.

Quick bootstrap check

# After installing sc, verify what's now available
sc plugins list 2>&1 | grep -iE "rtk|tokf|context|token|codefetch|project|yek|probe|colgrep|codedb|sloc|chainlink"

---

Quick Decision Tree

┌─ Am I about to run a command with verbose output? ─────→ rtk <command>
├─ Am I analyzing 50+ files or large output? ───────────→ context-mode
├─ Do I need to understand a codebase quickly? ─────────→ project-map → yek / codefetch
├─ Am I searching for code patterns? ───────────────────→ probe / colgrep / codedb
├─ Do I need token counts / bloat audit? ───────────────→ token-optimizer-cli / token-count
├─ Do I want automatic compression on any command? ─────→ tokf run <command>
├─ Am I about to run SQL queries? ─────────────────────→ slowql --non-interactive --export json
├─ Am I about to commit bloat-prone code? ──────────────→ sloc-guard check
├─ Am I resuming work across sessions? ─────────────────→ chainlink session start
├─ Nothing installed? ──────────────────────────────────→ See "Bare Metal" section above
└─ Do I need to discover optimization opportunities? ───→ sc discover --intent "token reduction"

Fallback decision tree (no tools at all)

┌─ Need to list a directory? ──────────────→ ls -m | head | sort
├─ Need to search for patterns? ────────────→ grep -rnl "pattern" --include="*.ext"
├─ Need file sizes? ───────────────────────→ du -sh * | sort -rh | head -20
├─ Need a codebase overview? ──────────────→ find . -type f -name "*.ext" | sort
├─ Need to count tokens/lines? ────────────→ wc -l -w -c
├─ Need compact git status? ───────────────→ git diff --stat
└─ Need to install real tools? ────────────→ sc plugins install <name>

Tier 1 — Primary Token Reduction

1. rtk (Rust Token Killer)

CLI proxy that intercepts command output and applies smart filtering.

Verified savings: 11.3M tokens saved at 86% efficiency across 4,439 commands.

# Install
rtk --version
# Hook setup for automatic savings
rtk init -g

# Core usage
rtk ls                    # Directory listing (compact)
rtk git status            # Compact git status
rtk git diff              # Ultra-condensed diff
rtk cargo build           # Build output (filtered)
rtk npm test              # Test output (failures only)
rtk tsc                   # TypeScript (grouped errors)
rtk grep <pattern>        # Compact grep

# Analytics
rtk gain                  # Token savings report
rtk gain --graph          # With visualization
rtk discover              # Find missed savings

How it works: Filters noise (comments, whitespace, boilerplate), groups similar items, truncates redundant context, deduplicates repeated log lines. Sub-5ms overhead per command.

2. tokf (Token Filter)

Config-driven CLI tool that compresses command output before it reaches LLM context. 51 built-in filters for common tools (docker, kubectl, git, npm, cargo, etc.).

# Install
cargo install tokf

# Core usage
tokf run <command>        # Run command with automatic filtering
tokf ls                   # List available filters
tokf gain                 # View token savings stats
tokf install <name>       # Install community filter
tokf setup                # Auto-detect tools and install hooks

# Filter management
tokf which <command>      # See which filter applies
tokf show <filter>        # View filter source
tokf verify               # Test filters work correctly

When to use tokf vs rtk:

• rtk for general-purpose command proxying (ls, git, build, test)
• tokf when you need specific tool-aware filtering (docker build, kubectl get pods, prisma generate) or custom configurable filter chains

3. context-mode (MCP Sandbox)

MCP server for large codebase analysis. Code runs in a sandboxed subprocess — only the summary enters context, achieving up to 98% token reduction.

# Install
npm install -g context-mode

# Core tools
# ctx_batch_execute — run multiple commands + search in one call
# ctx_execute — run code in sandbox, print only results
# ctx_execute_file — process a file in sandbox
# ctx_index + ctx_search — index content then query efficiently
# ctx_fetch_and_index — fetch URL, index, and search (HTML never enters context)

# Usage via stdio JSON-RPC
echo '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"ctx_execute","arguments":{"language":"shell","code":"echo hello"}}}' | context-mode 2>/dev/null

When to use context-mode:

• Analyzing 50+ files or output >1000 lines
• API calls, test runners, git queries with large output
• Repeated searches over the same content
• URL fetching (raw HTML never enters context)
• Data processing tasks

Tier 2 — Codebase Optimization

5. token-optimizer-cli

Scan codebases for token usage, detect bloat, get optimization suggestions. Supports GPT-4, GPT-4o, and Claude token counting.

# Scan all files
token-optimizer-cli scan . --json

# Deep audit with bloat detection
token-optimizer-cli audit . --json

# Single file check
token-optimizer-cli check file.ts

# Project-level summary
token-optimizer-cli summary . --json

6. token-count

Quick token/word/char/line counting. Pipe-compatible.

echo "text to count" | token-count
token-count < file.txt
cat code.ts | token-count

7. codefetch

Turn entire codebases into AI-friendly Markdown with token counting and tree views. Supports GPT-4 and Claude token encoders.

# Generate codebase markdown with token limit
codefetch --output analysis.md --max-tokens 5000

# With project tree
codefetch --project-tree 3 --max-tokens 3000

# Filter by extension
codefetch -e .ts,.js --exclude-dir node_modules

# Fetch from GitHub and analyze
codefetch --url https://github.com/user/repo --branch main

8. project-map

Instant codebase understanding — directory tree with file metadata in JSON. Essential first step before any deep dive.

# Overview
project-map
project-map --max-depth 3
project-map /path/to/project

9. yek

Fast Rust tool to serialize text files for LLM consumption. Configurable chunk sizing with token mode support and directory tree headers.

# Serialize to stdout with tree header
yek -t src/

# Token mode (split by tokens instead of bytes)
yek --tokens 8000 src/

# With line numbers and JSON output
yek --line-numbers --json src/

Tier 3 — Search Efficiency

10. probe

Semantic code search with AST parsing, token limits, and code extraction.

# Semantic search with token control
probe search "error handling" --max-tokens 1000

# Extract code blocks
probe extract file.ts --language typescript

# List symbols
probe symbols src/ --language rust

# AST pattern matching
probe query "function *($$$ARGS)" src/

11. colgrep

Semantic grep — find code by meaning, not just text. Natural language queries, hybrid search (grep + semantic ranking), 18+ languages.

# Natural language search
colgrep "authentication logic"
colgrep "database connection pool"

# Hybrid mode (grep + semantic)
colgrep -e "error" "error handling strategy"

# Limit results
colgrep "API endpoints" -k 5

12. codedb

Fast code intelligence — tree views, symbol search, trigram full-text search, dependency graphs, and O(1) word index. Zig-based CLI.

# Tree view
codedb tree
codedb tree --depth 3

# Hot files (most recently modified)
codedb hot

# Symbol search
codedb find function_name
codedb find ClassName

# Full-text search
codedb grep "search term"

Tier 4 — Bloat Prevention & Query Health

13. slowql — SQL Static Analyzer

Analyze SQL queries for performance, security, and compliance issues before they hit production. Catches bad SQL that would waste execution tokens on slow queries, full table scans, or security holes.

Verified: 272 rules, CI-pipeline-ready with --non-interactive and --export json.

# Install
pipx install slowql

# Analyze a SQL file (non-interactive, JSON export)
slowql --non-interactive --export json --input-file queries.sql

# Analyze stdin
cat queries.sql | slowql --non-interactive --export json

# Quick analysis
slowql --fast --input-file queries.sql

When to use slowql:

• Before running expensive queries that waste execution tokens
• In CI/CD pipelines to catch bad SQL patterns
• During code review of database-heavy PRs
• Proactive: prevents token waste at the query level

---

14. sloc-guard

Enforce source lines of code limits to prevent bloat. Git-aware, supports baseline grandfathering, trend tracking, and CI/CD integration.

# Check current SLOC against limits
sloc-guard check

# View trends
sloc-guard stats trend
sloc-guard stats summary

# Setup baseline
sloc-guard init --limit 1000

Tier 5 — Session & Context Management

15. chainlink

Issue tracking designed for AI agents. Track tasks, preserve context across sessions, leave handoff notes, and manage milestones. Local SQLite, zero auth.

# Start a focused session
chainlink session start

# Track progress
chainlink session action "Analyzed module X"
chainlink issue create "Refactor Y" -p high

# End with handoff
chainlink session end --notes "Ready for review"

# Resume later
chainlink session resume

Installation & Setup

One-time setup (ordered by priority)

# 1. RTK — instant savings on every command
rtk init -g
# Verify: rtk gain

# 2. tokf — tool-aware filtering
cargo install tokf
# Verify: tokf gain

# 3. context-mode — large codebase analysis
npm install -g context-mode
# Verify: context-mode --version

# 4. slowql — SQL static analysis
pipx install slowql
# Verify: slowql --help

# 5. token-optimizer-cli — periodic audits
token-optimizer-cli audit . --json | head

# 6. Verify all tools
rtk --version
tokf --version
slowql --help 2>&1 | head -1
token-optimizer-cli --help 2>&1 | head -2
context-mode --version 2>&1 | head -1
project-map --help 2>&1 | head -2
yek --version
probe --help 2>&1 | head -2
colgrep --help 2>&1 | head -2
codedb --help 2>&1 | head -2
sloc-guard --help 2>&1 | head -2
chainlink --help 2>&1 | head -2

Workflow Patterns

Daily Development — Maximum Token Savings

# 1. Resume context
chainlink session start

# 2. Understand current state
project-map --max-depth 2
rtk git status          # Compressed git status
rtk git diff

# 3. Search efficiently
probe search "relevant feature" --max-tokens 500
colgrep "target pattern" -k 5
codedb find relevant_symbol

# 4. Build/test with compression
rtk cargo build
rtk cargo test
tokf run docker-compose up -d

# 5. Large analysis
# Use context-mode: ctx_batch_execute or ctx_execute
# Instead of reading 20 files into context

# 6. Track progress
chainlink session action "Built feature X"
chainlink session end --notes "All tests passing"

Large Codebase Onboarding

# 1. Map the terrain
project-map

# 2. Serialize with token control
yek -t --tokens 5000 src/

# 3. Generate codebase markdown
codefetch --project-tree 3 --max-tokens 10000

# 4. Audit token hotspots
token-optimizer-cli audit . --json | grep -E "warning|bloat"

# 5. Dive selectively with smart search
probe symbols src/ --language typescript
codedb find main_entry
colgrep "core functionality" -k 10

Pre-Commit Quality Gate

# 1. Check for bloat
sloc-guard check

# 2. Audit token impact of changes
rtk git diff
slowql --non-interactive --export json --input-file migrations/  # Check SQL changes
token-optimizer-cli check src/changed-file.ts

# 3. Structural search for bad patterns
probe query "console.log($$$ARGS)" src/ --language javascript

# 4. Track via chainlink
chainlink issue create "Weekly code quality pass"

Debugging / Bug Investigation

chainlink session start
chainlink issue create "Bug: X" -p critical -l bug
colgrep "error handling" -k 10
probe search "failure pattern" --max-tokens 800
codedb find error_handler
rtk git log -n 20       # Check recent commits for what changed
rtk grep "ERROR" logs/
slowql --non-interactive --fast --input-file suspect.sql  # Check if SQL is the issue
context-mode ctx_execute: analyze logs in sandbox
chainlink session end --notes "Root cause identified"

Tool Selection Matrix

Situation	Tool	Why
Running any command	rtk	60-90% automatic savings
Docker/kubectl/prisma	tokf	Custom filters for tool output
50+ file analysis	context-mode	98% reduction via sandbox
Codebase overview	project-map	Instant, token-efficient
Codebase dump for LLM	yek / codefetch	Token-controlled serialization
Semantic search	probe / colgrep	Avoid grep bloat, AST-aware
Symbol lookup	codedb	O(1) word index
SQL query analysis	slowql	Catch bad SQL before it wastes tokens
Token bloat audit	token-optimizer-cli	Finds worst offenders
Pre-commit quality	sloc-guard	Prevents bloat accumulation
Multi-session work	chainlink	Preserve context, handoff
Quick token count	token-count	Pipe-friendly, instant
Discover new tools	sc discover	Find optimization opportunities

Token Savings Benchmarks

All benchmarks verified from live system:

• rtk: 86% average savings on 4,439 commands (11.3M tokens saved)
• context-mode: Up to 98% reduction vs reading files directly
• tokf: 51 built-in filters for common development tools
• slowql: 272 analysis rules, catches bad SQL before token waste
• token-optimizer-cli: Scanned 11,814 files, detected 1,733 bloat files
• project-map: Fixed 2-depth tree = ~200 tokens vs reading directory structure raw
• yek: Token-mode serialization avoids overflowing context with irrelevant files

Health Check

Run periodically to ensure all tools are operational:

echo "=== TOKEN REDUCTION HEALTH CHECK ==="
echo ""
echo "--- rtk ---"
rtk --version 2>&1 && echo "✓ rtk installed"
echo ""

echo "--- tokf ---"
tokf --version 2>&1 && echo "✓ tokf installed"
echo ""

echo "--- context-mode ---"
context-mode --version 2>&1 | head -1 && echo "✓ context-mode installed"
echo ""

echo "--- slowql ---"
slowql --help 2>&1 | head -1
echo ""

echo "--- token-optimizer-cli ---"
token-optimizer-cli --help 2>&1 | head -1
echo ""

echo "--- project-map ---"
project-map --help 2>&1 | head -1
echo ""

echo "--- yek ---"
yek --version
echo ""

echo "--- probe ---"
probe --help 2>&1 | head -1
echo ""

echo "--- colgrep ---"
colgrep -- 2>&1 |  -1
 

 
codedb -- 2>&1 |  -1
 

 
sloc-guard -- 2>&1 |  -1
 

 
chainlink -- 2>&1 |  -1
 

 
token-count -- 2>&1 |  -1
 

 
rtk gain 2>&1 |  -10
 

 
token-optimizer-cli summary . --json 2>&1 |  -5
 
 
 
sc discover --intent  2>&1 | python3 -c  2>&1
 
 

Continuous Optimization

Weekly:

rtk gain              # Check savings trend
tokf gain             # Check tokf usage
slowql --non-interactive --fast --input-file recent.sql  # Check for SQL issues
token-optimizer-cli audit . --json  # Scan for new bloat

Monthly:

sc discover --intent "token reduction"  # Find new plugins
sc plugins explore | grep -iE "token|compress|context"  # Browse for updates
sloc-guard stats trend  # Track SLOC trends

When starting a new project:

project-map
yek -t --tokens 5000 .
token-optimizer-cli scan . --json
chainlink session start

Verification

This skill was built by:

1. Discovering tools via sc bin / sc discover --intent "token reduction"
2. Installing and smoke-testing each tool
3. Verifying real-world savings (rtk: 11.3M tokens, 86%)
4. Testing each command pattern in the skill
5. Combining complementary tools into unified workflows