Netflix engineer Tejas Chopra creates Headroom, an open-source tool that compresses AI prompts to reduce token costs, and estimates it has saved users $700,000

Chopra, a Netflix senior engineer, built Headroom to prune redundant tokens from prompts before they reach language models. He estimates that as much as 90% of tokens are redundant. The tool, not an official Netflix project but already used by several Netflix teams and external projects, has been available since January and currently stands at v0.22, with 2,000 GitHub stars and over 120 forks.

Headroom runs as a proxy on a developer's machine and uses multiple compression techniques: CacheAligner detects unchanged information to avoid cache misses; a router sends content to specialized compressors (Abstract Syntax Tree for code, JSON and DOM compressors for boilerplate); and 'squashers' use statistical analysis to identify relevant text. A final process called Compress Cache and Retrieve allows the model to retrieve original uncompressed data from Redis or SQLite if needed.

Headroom users collectively now have 200 billion tokens to spend elsewhere, and Chopra said 'A lot of our users are people who have been really burned by token costs, more than anything else.' Research suggests that reducing context can both save costs and improve model performance: a Stanford study found that LLMs pay more attention to the beginning and end of the context window and disregard the middle, while researchers from Chroma found that 'performance grows increasingly unreliable as input length grows' across 18 LLMs.