AIToday
Large Language ModelsAI Coding AssistantsOpen-Source AIAmazon AI BlogPublished: Nov 21, 2026, 13:00 JST

AWS adds distributed KV caching skill to Agent Toolkit for AWS

Send on LINE
AWS adds distributed KV caching skill to Agent Toolkit for AWS

3 Key Points

  1. What happened

    AWS added a distributed KV caching skill to its Agent Toolkit for AWS, letting coding agents like Kiro and Claude Code check, size and set up an Amazon ElastiCache cache for self-hosted LLM fleets.

  2. Why it matters

    The skill lets coding agents evaluate and configure a shared cache tier from a few prompts, potentially reducing redundant work across servers for self-hosted LLM deployments.

WHO IT HITSTeams self-hosting LLMs on GPUs, such as those running vLLM or SGLang fleets, can use the skill to assess whether a shared cache pays off before deploying infrastructure.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The skill is positioned as a way to avoid redundant GPU work when self-hosted LLMs resend large contexts. A legal AI assistant with hundreds of thousands of tokens of case history exemplifies the problem: when a lawyer returns to a thread, the GPU server that picks up the request recomputes key-value attention states for all tokens before responding. AWS says distributed KV caching with Amazon ElastiCache can cut self-hosted LLM inference costs by up to 40% and time to first token by up to 80%. The skill guides a coding agent through four flows: diagnosing a slow first token, sizing and pricing the cache, setting it up, and troubleshooting a failed transfer. Before any spend, the agent performs a fit check by reading the GPU fleet with a read-only describe-instances call and counting EFA interfaces. It then asks only for details it cannot see, such as engine, model, tensor-parallel size, KV data type, server price, and prompt repetition frequency. For sizing, the agent uses KVTrace to estimate savings from the customer's own traffic at no additional charge, and stops if the savings do not cover the cache's cost. The skill also implements best practices, including emptying local tiers and checking that two cache-loaded runs give the same output before production traffic, and seeking explicit approval for high-impact operations.

FAQ
Which coding agents can use this skill?
The skill works with Kiro, Claude Code, OpenAI Codex, and other coding agents that support Agent Skills.
What prerequisites are needed to set up the cache?
You need a coding agent that supports Agent Skills, GPU instances that support EFA RDMA write such as p5, LMCache 0.5.8rc1 or later, vLLM 0.11.1 or later (or SGLang 0.5.13 or later), and ElastiCache for Valkey 9.1 or later on cache.i8ge.48xlarge nodes.
How does the skill estimate savings before deployment?
The skill uses KVTrace to estimate savings from your own workload traffic at no additional charge, and stops if the savings don't cover the cache's cost.
Amazon AI BlogRead Original Article
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleDell at New York Symposium: AI moat is your own data, not models