AWS shows task-aware compression cuts AI query costs by up to 98%, beating RAG

AWS published an open-source implementation of task-aware knowledge compression (TAKC), a technique designed to overcome limitations in Retrieval-Augmented Generation (RAG) for complex enterprise analytical tasks. The problem TAKC targets is concrete: when a private equity firm conducts due diligence on a $500 million（約800億円） acquisition of a manufacturing company, the team must analyze financial statements spanning 12 subsidiaries over 5 years, 200+ supplier contracts, environmental compliance reports from 8 facilities, and 50+ legal cases. When an analyst asks about consolidated financial risks given current supplier terms and pending litigation, RAG's similarity search cannot surface the answer because the relevant information is scattered across hundreds of documents and the connections between them share no lexical similarity.

TAKC works by using an LLM to produce shorter, task-focused summaries of documents during an offline ingestion phase. The key innovation is that different tasks require different information from the same document: an annual report compressed for financial analysis preserves revenue figures, margins, and cash flow data, while the same report compressed for a compliance review preserves regulatory citations and violation histories. The system maintains four compression tiers for each task type—light (8x), medium (16x), high (32x), and ultra (64x)—each reducing context by approximately 87.5%, 93.8%, 96.9%, and 98.4% respectively. At query time, a complexity analyzer routes incoming questions to the appropriate tier based on signals like query length and question type. Simple factual questions hit the ultra-compressed cache while complex analytical questions use the lightly compressed cache. The compression reduces token count by 8x to 64x while targeting task-relevant information for retention.

For a 100,000-token knowledge base queried 1,000 times per day, this translates to significant cost savings: full context costs 100,000,000 daily input tokens (100% relative cost), RAG's top-10 chunks cost approximately 10,000,000 tokens (10%), while TAKC tiers range from approximately 12,500,000 tokens (12.5% at 8x) down to approximately 1,563,000 tokens (1.6% at 64x). The implementation deploys on AWS using AWS Lambda for compute, Amazon ElastiCache Serverless for caching, Amazon Bedrock for compression and inference, Amazon S3 for storage, Amazon API Gateway and Amazon Cognito for authentication, and AWS WAF for security. The ingestion pipeline chunks documents into 256-token segments with 50-token overlap, then invokes a compression Lambda function that calls Amazon Bedrock at all four compression tiers using a task-aware prompt that specifies exactly what information to preserve. Compressed outputs are stored in ElastiCache with a 24-hour TTL and backed up to S3; if a cache entry is evicted or expires, the query function falls back to the S3 backup and re-populates the cache on read. The entire infrastructure is defined as a single AWS Cloud Development Kit stack and deploys with a single command.

The trade-off is that TAKC requires upfront compression cost through one-time Bedrock calls during ingestion, which amortizes only for knowledge bases that change infrequently and get queried repeatedly. For knowledge bases that change hourly, RAG's per-query retrieval model may be more practical. AWS recommends TAKC for cross-document reasoning and synthesis on well-defined task types with stable knowledge bases, and RAG for narrow factual lookups, unpredictable query patterns, or hourly-changing content.

Task-aware knowledge compression addresses a real ceiling in RAG for enterprise analytical work. When an analyst at a private equity firm evaluates a $500 million（約800億円） acquisition and must synthesize financial risks, supplier terms, and pending litigation across 12 subsidiaries, 200+ contracts, and 50+ legal cases, RAG's lexical similarity search fails because the relevant connections share no vocabulary overlap. TAKC solves this by compressing the entire corpus through the lens of the task—financial analysis, legal risk review, compliance audit—and preserving task-relevant relationships that RAG would discard.

The four-tier compression strategy (8x to 64x) reflects a practical insight: most enterprise queries are simple lookups ("What was Q3 revenue?") that need minimal context, while complex analytical questions need richer detail. A query complexity analyzer routes straightforward questions to ultra-compressed (64x) tiers at minimal cost, reserving the lightly compressed (8x) cache for cross-document synthesis. This tier-based routing complements existing RAG optimizations like metadata filtering and query reformatting.

Cost amortization is the key constraint. TAKC requires one-time Bedrock compression calls during ingestion—a significant upfront cost—which pays off only for knowledge bases queried repeatedly with infrequent changes. Knowledge bases that change hourly remain better served by RAG's per-query retrieval model, where the compression cost per document is paid once and cached, then re-executed only when content changes.

No comments yet. Be the first to share your thoughts!