Token Optimizer

Token Optimizer MCP

Intelligent token optimization through caching, compression, and smart tooling for Claude Code and Claude Desktop

Overview

Token Optimizer MCP is a Model Context Protocol (MCP) server that reduces context window usage by 60-90% through intelligent caching, compression, and smart tool replacements. By storing compressed content externally in SQLite and providing optimized alternatives to standard tools, the server helps you maximize your available context window.

Production Results: 60-90% token reduction across 38,000+ operations in real-world usage.

Key Features

• Smart Tool Replacements: Automatic optimization for Read, Grep, Glob, and more
• Context Window Optimization: Store content externally to free up context space
• High Compression: Brotli compression (2-4x typical, up to 82x for repetitive content)
• Persistent Caching: SQLite-based cache that persists across sessions
• Accurate Token Counting: Uses tiktoken for precise token measurements
• 61 Specialized Tools: File operations, API caching, database optimization, monitoring, and more
• Zero External Dependencies: Completely offline operation
• Production Ready: Built with TypeScript for reliability

Installation

Quick Install (Recommended)

Windows

# Run PowerShell as Administrator, then:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Install globally (hooks install automatically!)
npm install -g @ooples/token-optimizer-mcp

macOS / Linux

# Install globally (hooks install automatically!)
npm install -g @ooples/token-optimizer-mcp

That's it! The postinstall script will automatically:

1. ✅ Install token-optimizer-mcp globally via npm
2. ✅ Auto-detect and configure all installed AI tools (Claude Desktop, Cursor, Cline, etc.)
3. ✅ Set up automatic token optimization on every tool call
4. ✅ Configure workspace trust and execution permissions

Result: 60-90% token reduction across all operations!

Note: If automatic installation is skipped (e.g., in CI environments), you can manually run the installer:

• Windows: powershell -ExecutionPolicy Bypass -File install-hooks.ps1
• macOS/Linux: bash install-hooks.sh

Manual Configuration

For detailed platform-specific installation instructions, see docs/HOOKS-INSTALLATION.md.

Available Tools (65 Total)

Core Caching & Optimization (8 tools)

• optimize_text - Compress and cache text (primary tool for token reduction)
• get_cached - Retrieve previously cached text
• compress_text - Compress text using Brotli
• decompress_text - Decompress Brotli-compressed text
• count_tokens - Count tokens using tiktoken (GPT-4 tokenizer)
• analyze_optimization - Analyze text and get optimization recommendations
• get_cache_stats - View cache hit rates and compression ratios
• clear_cache - Clear all cached data

Usage Example:

// Cache large content to remove it from context window
optimize_text({
  text: "Large API response or file content...",
  key: "api-response-key",
  quality: 11
})
// Result: 60-90% token reduction

Smart File Operations (10 tools)

Optimized replacements for standard file tools with intelligent caching and diff-based updates:

• smart_read - Read files with 80% token reduction through caching and diffs
• smart_write - Write files with verification and change tracking
• smart_edit - Line-based file editing with diff-only output (90% reduction)
• smart_grep - Search file contents with match-only output (80% reduction)
• smart_glob - File pattern matching with path-only results (75% reduction)
• smart_diff - Git diffs with diff-only output (85% reduction)
• smart_branch - Git branch listing with structured JSON (60% reduction)
• smart_log - Git commit history with smart filtering (75% reduction)
• smart_merge - Git merge management with conflict analysis (80% reduction)
• smart_status - Git status with status-only output (70% reduction)

Usage Example:

// Read a file with automatic caching
smart_read({ path: "/path/to/file.ts" })
// First read: full content
// Subsequent reads: only diff (80% reduction)

API & Database Operations (10 tools)

Intelligent caching and optimization for external data sources:

• smart_api_fetch - HTTP requests with caching and retry logic (83% reduction on cache hits)
• smart_cache_api - API response caching with TTL/ETag/event-based strategies
• smart_database - Database queries with connection pooling and caching (83% reduction)
• smart_sql - SQL query analysis with optimization suggestions (83% reduction)
• smart_schema - Database schema analysis with intelligent caching
• smart_graphql - GraphQL query optimization with complexity analysis (83% reduction)
• smart_rest - REST API analysis with endpoint discovery (83% reduction)
• smart_orm - ORM query optimization with N+1 detection (83% reduction)
• smart_migration - Database migration tracking (83% reduction)
• smart_websocket - WebSocket connection management with message tracking

Usage Example:

// Fetch API with automatic caching
smart_api_fetch({
  method: "GET",
  url: "https://api.example.com/data",
  ttl: 300
})
// Cached responses: 95% token reduction

Build & Test Operations (10 tools)

Development workflow optimization with intelligent caching:

• smart_build - TypeScript builds with diff-based change detection
• smart_test - Test execution with incremental test selection
• smart_lint - ESLint with incremental analysis and auto-fix
• smart_typecheck - TypeScript type checking with caching
• smart_install - Package installation with dependency analysis
• smart_docker - Docker operations with layer analysis
• smart_logs - Log aggregation with pattern filtering
• smart_network - Network diagnostics with anomaly detection
• smart_processes - Process monitoring with resource tracking
• smart_system_metrics - System resource monitoring with performance recommendations

Usage Example:

// Run tests with caching
smart_test({
  onlyChanged: true,  // Only test changed files
  coverage: true
})

Advanced Caching (10 tools)

Enterprise-grade caching strategies with 87-92% token reduction:

• smart_cache - Multi-tier cache (L1/L2/L3) with 6 eviction strategies (90% reduction)
• cache_warmup - Intelligent cache pre-warming with schedule support (87% reduction)
• cache_analytics - Real-time dashboards and trend analysis (88% reduction)
• cache_benchmark - Performance testing and strategy comparison (89% reduction)
• cache_compression - 6 compression algorithms with adaptive selection (89% reduction)
• cache_invalidation - Dependency tracking and pattern-based invalidation (88% reduction)
• cache_optimizer - ML-based recommendations and bottleneck detection (89% reduction)
• cache_partition - Sharding and consistent hashing (87% reduction)
• cache_replication - Distributed replication with conflict resolution (88% reduction)
• predictive_cache - ML-based predictive caching with ARIMA/LSTM (91% reduction)

Usage Example:

// Configure multi-tier cache
smart_cache({
  operation: "configure",
  evictionStrategy: "LRU",
  l1MaxSize: 1000,
  l2MaxSize: 10000
})

Monitoring & Dashboards (7 tools)

Comprehensive monitoring with 88-92% token reduction through intelligent caching:

• alert_manager - Multi-channel alerting (email, Slack, webhook) with routing (89% reduction)
• metric_collector - Time-series metrics with multi-source support (88% reduction)
• monitoring_integration - External platform integration (Prometheus, Grafana, Datadog) (87% reduction)
• custom_widget - Dashboard widgets with template caching (88% reduction)
• data_visualizer - Interactive visualizations with SVG optimization (92% reduction)
• health_monitor - System health checks with state compression (91% reduction)
• log_dashboard - Log analysis with pattern detection (90% reduction)

Usage Example:

// Create an alert
alert_manager({
  operation: "create-alert",
  alertName: "high-cpu-usage",
  channels: ["slack", "email"],
  threshold: { type: "above", value: 80 }
})

System Operations (6 tools)

System-level operations with smart caching:

• smart_cron - Scheduled task management (cron/Windows Task Scheduler) (85% reduction)
• smart_user - User and permission management across platforms (86% reduction)
• smart_ast_grep - Structural code search with AST indexing (83% reduction)
• get_session_stats - Session-level token usage statistics
• analyze_project_tokens - Project-wide token analysis and cost estimation
• optimize_session - Compress large file operations from current session

Usage Example:

// View session token usage
get_session_stats({})
// Result: Detailed breakdown of token usage by tool

How It Works

Token Analytics (4 tools)

Granular token usage analytics for pinpointing optimization opportunities:

• get_hook_analytics - Token usage breakdown by hook phase (PreToolUse, PostToolUse, etc.)
• get_action_analytics - Token usage breakdown by tool/action (Read, Write, Grep, etc.)
• get_mcp_server_analytics - Token usage breakdown by MCP server (token-optimizer, filesystem, etc.)
• export_analytics - Export analytics data in JSON or CSV format with filtering

Usage Example:

// Get per-hook analytics
get_hook_analytics({
  startDate: "2025-01-01T00:00:00Z",
  endDate: "2025-12-31T23:59:59Z"
})
// Result: Shows which hooks consume the most tokens

// Get per-action analytics
get_action_analytics({})
// Result: Shows which tools use the most tokens

// Export analytics as CSV
export_analytics({
  format: "csv",
  hookPhase: "PreToolUse"
})
// Result: CSV export filtered by PreToolUse hook

Key Features:

• Per-hook phase tracking (PreToolUse, PostToolUse, SessionStart, etc.)
• Per-action tracking (Read, Write, count_tokens, etc.)
• Per-MCP-server tracking (token-optimizer, filesystem, GitHub, etc.)
• Date range filtering
• JSON and CSV export
• Persistent storage with SQLite
• Zero performance impact (async batched writes)

Global Hooks System (7-Phase Optimization)

When global hooks are installed, token-optimizer-mcp runs automatically on every tool call:

┌─────────────────────────────────────────────────────────────┐
│ Phase 1: PreToolUse - Tool Replacement                      │
│ ├─ Read   → smart_read   (80% token reduction)             │
│ ├─ Grep   → smart_grep   (80% token reduction)             │
│ └─ Glob   → smart_glob   (75% token reduction)             │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 2: Input Validation - Cache Lookups                   │
│ └─ get_cached checks if operation was already done          │
└─────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 3: PostToolUse - Output Optimization                  │
│ ├─ optimize_text for large outputs                          │
│ └─ compress_text for repeated content                       │


…