Cutting AI API Costs 60–90% With a Token Efficiency Proxy

If you run AI agents at any real volume, token costs compound fast. We built a lightweight proxy layer that sits between our AI tools and their commands, filtering out redundant output before it ever reaches the language model. The result: 60–90% token savings on common development operations with zero change to the underlying workflow.

Where Tokens Actually Go

We audited our AI session logs and found that a significant fraction of context was consumed by verbose command output that provided no signal: full directory listings when only the first few files mattered, complete diff output when only changed line counts were needed, raw JSON API responses with dozens of irrelevant fields, full file contents when only a specific section was requested.

These aren’t edge cases—they’re the default behavior of most CLI tools. A git log with no arguments returns the full commit history. An ls on a large directory returns hundreds of filenames. Left unfiltered, this output fills the context window with noise.

The Proxy Approach

The solution was a transparent proxy that intercepts common commands, runs them normally, then filters the output down to what’s actually useful before returning it to the AI. From the AI’s perspective, it ran the command and got a result. In practice, the result has been trimmed to its essential content.

The filtering rules are per-command and tunable:

• Directory listings: show first N files, summarize the rest (“…and 847 more”)
• Git log: show commit hashes and first line of messages only
• Git diff: show changed file names and line counts, omit full patch unless requested
• JSON output: extract and surface only tagged or commonly-useful fields

The Self-Auditing Agent

Beyond the proxy, we deployed a weekly agent (Whetstone) that analyzes session logs to identify new sources of token waste that the proxy doesn’t yet cover. It scores each candidate by estimated savings versus implementation complexity, implements the top-scored fixes autonomously, and writes a report. Over time this compounds: the system gets more efficient each week without manual intervention.

The Practical Takeaway

Token efficiency is infrastructure work, not afterthought. Every organization running AI at volume should audit what’s actually consuming context in their sessions. The answer is usually not the AI’s reasoning—it’s the raw, unfiltered data being fed to it. Filtering that data at the source is almost always the highest-ROI optimization available.