AWS adds rate limiting to Bedrock AgentCore gateway

Amazon Bedrock AgentCore gateway is a fully managed, serverless AI gateway that routes traffic to tools (managed web search, managed knowledge bases, MCP servers), inference models (LLMs), agents, and HTTP endpoints. Today Amazon announced support for rate limiting on this gateway, enabling organizations to define per-user and per-group traffic controls.

Rate limiting operates on three dimensions. Request rate limits, measured in requests per second (RPS) and requests per minute (RPM), apply to all target types; each request counts as one unit regardless of how long it takes to complete—a 50-millisecond request and a 90-second streaming request each consume exactly one unit. Token rate limits, measured in tokens per minute (TPM), apply to inference targets only and account for both input and output tokens; the gateway estimates incoming tokens upfront using a general-purpose tokenizer and later reconciles the limit with actual token usage reported by the model provider. Connection rate limits, measured in connections per second (CPS), apply to all target types and track how long each request holds an open connection; a 100-second streaming inference call consumes one connection slot for its entire duration, providing protection against long-lived concurrent sessions.

Rate limit configurations consist of dimension keys (which group traffic into buckets) and entries (which define the allowed throughput for each bucket). Dimension keys include targetName, toolName, qualifiedModelId, JWT claims, IAM principals, and IAM source identity. Entries specify a set of dimension keys to match and the allowed throughput; they support a wildcard default value (*) that gives each distinct value its own independent bucket. When a request arrives, the gateway resolves each dimension key to its value from the request context and checks whether a named entry matches before falling back to the wildcard; named entries take precedence.

The example configuration in the announcement uses three user groups (Basic, Advanced, and Beta) authenticated via JWT with Microsoft Entra ID as the identity provider. AgentCore Identity enforces role-based access control (RBAC), scoping each group's access to specific targets and models. A per-group rate limit assigns Basic users 100 RPM and 50 CPS, Advanced users 300 RPM and 150 CPS, and users with both Advanced and Beta roles 300 RPM and 200 CPS (the higher connection allowance accommodates streaming-heavy benchmarking workloads). However, within each group, a single user can consume the entire group quota, throttling peers. To prevent this, a second per-user rate limit configuration uses both role and JWT subject ($.context.jwt.sub) as dimension keys, capping each individual Basic user at 20 RPM and 10 CPS, each Advanced user at 60 RPM and 30 CPS, and Advanced–Beta users at 60 RPM and 50 CPS. Both rate limits are evaluated independently using AND logic: a request must pass both the group-level limit and the per-user limit to proceed.

Rate limiting is enforced in two layers. Customer-defined rate limits are evaluated first; if the request passes, service quotas (managed by AWS and enforced per account) are evaluated. The effective rate for requests is the minimum of the customer-defined limit and the service-managed limit. Some service quotas can be increased via the Service Quotas console. This architecture ensures that no customer-defined policy can exceed the account-level safety ceiling, while organizations retain full flexibility to implement fair-usage policies tailored to their user base and workload characteristics.