Imported from wcz234/skills-token-optimizer (SKILL.md). Install upstream with npx skills add wcz234/skills-token-optimizer. Copyright stays with the author.

Token Optimizer Skill

Minimize LLM token consumption through intelligent output filtering and compression

This skill applies RTK-style filtering strategies to reduce token usage by 60-90% while preserving essential information.

Core Principles

1. Filter before sending - Compress output before it reaches the LLM
2. Preserve signal, remove noise - Keep errors, failures, and changes; hide passing tests, progress bars, and boilerplate
3. Progressive disclosure - Show summary by default, provide full output on demand
4. Track savings - Record token savings for analytics

Filtering Strategies

1. Stats Extraction (90-99% reduction)

For commands that produce verbose success output:

# Instead of full git status output (200 tokens)
# Send: "3 modified, 1 untracked ✓" (~10 tokens)

# Instead of full test output (5000 tokens)
# Send: "✓ 18/20 tests passed, 2 failed" (~8 tokens)

Apply to: git status, git log, git diff, test runners, build tools

2. Error-Only Mode (60-80% reduction)

For build/test/lint commands:

# Show only failures and warnings
# Hide: passing tests, successful builds, info messages

# Example output:
❌ test_auth_login: AssertionError at line 42
⚠️ 3 deprecation warnings
✓ 47/50 tests passed

Apply to: cargo test, npm test, pytest, tsc, eslint

3. Grouping & Aggregation (80-90% reduction)

Group similar items:

# Instead of listing 100 lint errors individually
# Send:
no-unused-vars: 23 occurrences
  - src/utils.ts:5,12
  - src/api.ts:8
semi: 45 occurrences
  (truncated, see full log)

Apply to: linters, type checkers, search results

4. Tree Compression (50-70% reduction)

Compress directory listings:

# Instead of flat ls -la (800 tokens)
# Send:
src/ (8 files, 12KB)
  ├─ main.rs
  ├─ utils.rs
  └─ lib/ (12 files)
tests/ (5 files)
Cargo.toml
README.md

Apply to: ls, tree, find

5. Code Filtering (60-90% reduction)

Strip non-essential code:

// FilterLevel::Minimal - Strip comments only
fn calculate_total(items: &[Item]) -> i32 {
    items.iter().map(|i| i.value).sum()
}

// FilterLevel::Aggressive - Strip function bodies
fn calculate_total(items: &[Item]) -> i32 { ... }

Apply to: cat, read, code review

6. Progress Filtering (85-95% reduction)

Strip progress bars and live updates:

# Instead of animated progress:
# [====================> ] 85%  00:02:15
# Sending: "✓ Downloaded 125MB"

Apply to: wget, curl, npm install, docker pull

7. Deduplication (70-85% reduction)

Collapse repeated lines:

# Instead of 50 identical error lines:
# Send: "[ERROR] Connection timeout (×50)"

Apply to: logs, repeated errors

8. Structure-Only for JSON (80-95% reduction)

Show JSON schema without values:

// Instead of full JSON (1000 tokens)
// Send:
{
  "user": { "id": "...", "name": "...", "email": "..." },
  "posts": [ { "title": "...", "date": "..." }, ... ]
}

Apply to: JSON responses, config files

Command-Specific Filters

Git Operations

Test Runners

File Operations

Build Tools

Usage Patterns

Basic Usage

# Standard (auto-detects and applies optimal filter)
rtk git status
rtk cargo test
rtk ls -la

# Verbose (show full output)
rtk git status -v
rtk cargo test -vv

# Ultra-compact (maximum compression)
rtk git status -u
rtk ls -u

In LLM Context

When sending command output to LLM:

## Command Output (Filtered, 90% token savings)

✓ 18/20 tests passed ❌ 2 failed:

• test_auth_login: AssertionError at line 42
• test_parse: panic at utils.rs:18

[Full log: ~/.local/share/rtk/tee/1707753600.log]

Tee Mechanism (Full Output Recovery)

When a command fails or user requests verbose mode, save full output:

# Automatically saved on failure
~/.local/share/rtk/tee/<timestamp>_<command>.log

# LLM can reference without re-executing
[Full output: ~/.local/share/rtk/tee/1707753600_cargo_test.log]

Token Tracking

Track token savings over time:

# Summary statistics
rtk gain

# Output example:
Total commands: 1,247
Original tokens: 2.4M
Filtered tokens: 480K
Saved: 1.9M tokens (80%)
Avg savings per command: 1,560 tokens

# Graph (last 30 days)
rtk gain --graph

# Daily breakdown
rtk gain --daily

Implementation Guide

Creating a Filter Module

// Example: git_status_filter.rs
pub fn filter_git_status(output: &str) -> String {
    let mut modified = 0;
    let mut untracked = 0;
    let mut staged = 0;
    
    for line in output.lines() {
        if line.starts_with("M") { modified += 1; }
        if line.starts_with("??") { untracked += 1; }
        if line.starts_with("A") { staged += 1; }
    }
    
    format!(
        "{} modified, {} untracked, {} staged ✓",
        modified, untracked, staged
    )
}

Bash Script Version

#!/bin/bash
# scripts/filter-git-status.sh

output=$(git status --porcelain)
modified=$(echo "$output" | grep -c "^ M")
untracked=$(echo "$output" | grep -c "^??")
staged=$(echo "$output" | grep -c "^A")

echo "$modified modified, $untracked untracked, $staged staged ✓"

Best Practices

DO ✅

• Show summary statistics (counts, percentages)
• Highlight failures and errors
• Provide file paths for issues
• Include exit codes
• Offer full output path on demand
• Use color/icons for quick scanning

DON'T ❌

• Send full success logs
• Include progress bars
• Repeat identical lines
• Show boilerplate headers/footers
• Send ANSI escape codes unnecessarily
• Hide critical errors in verbose output

Verbosity Levels

Configuration

Optional config file: ~/.config/token-optimizer/config.toml

[tracking]
enabled = true
database_path = "~/.local/share/token-optimizer/history.db"

[filters]
default_level = "smart"  # smart, minimal, aggressive
save_tee_on_failure = true
max_tee_files = 20

[commands]
# Override default filters for specific commands
git_status = "stats"
cargo_test = "failures-only"
ls = "tree"

Examples

Example 1: Test Failure Analysis

Without optimization (5000 tokens):

running 50 tests
test test_auth_login ... ok
test test_auth_logout ... ok
... (47 more lines)
test test_parse ... FAILED
thread 'test_parse' panicked at 'assertion failed', src/utils.rs:18:5
...

With optimization (50 tokens):

❌ 2/50 tests failed

test_parse: panic at utils.rs:18
test_auth_login: AssertionError at line 42

[Full log: ~/.local/share/rtk/tee/1707753600.log]

Example 2: Directory Listing

Without optimization (800 tokens):

drwxr-xr-x 15 user staff 480 Jan 1 12:00 .
drwxr-xr-x 12 user staff 384 Jan 1 12:00 ..
-rw-r--r--  1 user staff  12K Jan 1 12:00 main.rs
... (45 more lines)

With optimization (100 tokens):

src/ (8 files, 45KB)
  ├─ main.rs (12K)
  ├─ utils.rs (8K)
  └─ lib/ (12 files)
tests/ (5 files)
Cargo.toml
README.md

Example 3: Git Diff

Without optimization (2000 tokens):

diff --git a/src/main.rs b/src/main.rs
index abc123..def456 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -15,7 +15,7 @@ fn main() {
-    println!("Hello");
+    println!("Hello, World!");
... (200 more lines)

With optimization (80 tokens):

3 files changed
  src/main.rs: +5, -2
  src/utils.rs: +12, -8
  tests/test_auth.rs: +8, -2

Total: +25, -12

Integration with LLM Agents

For Claude Code

Add to ~/.claude/settings.json:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "command": "token-opt-rewrite.sh"
      }
    ]
  }
}

For OpenCode

Create ~/.config/opencode/plugins/token-optimizer.ts:

export default {
  hooks: {
    'tool.execute.before': async (tool) => {
      if (tool.type === 'bash') {
        return { ...tool, command: `rtk ${tool.command}` };
      }
      return tool;
    }
  }
};

Metrics & Analytics

Track these metrics:

1. Token Savings: Original vs filtered tokens
2. Command Coverage: % of commands optimized
3. User Satisfaction: Did filter preserve needed info?
4. Performance: Filter overhead (<10ms target)

Troubleshooting

Filter removed important info

Solution: Use -v flag or check tee file

False positive on error detection

Solution: Adjust error patterns in filter module

Performance overhead too high

Solution: Simplify filter logic, use early returns

References

• RTK Architecture
• RTK Filtering Strategies
• Token Economics

License

MIT - Based on RTK architecture