Azure API Management gateway lets 800-token cap reach 1,472

The test centered on a single policy, llm-token-limit, which returns 403 when a period quota is exceeded and 429 when a per-minute rate is exceeded. Microsoft's own documentation notes that parallel execution can cause temporary overshoot, and this experiment set out to measure that range directly. The setup used an APIM Developer classic instance with one unit in Japan East, the gpt-4.1-mini model dated 2025-04-14 on Regional Standard, and the Chat Completions API dated 2024-10-21. Each trial used a fresh counter with a fixed synthetic prompt and max_tokens of 128, so every successful response consumed the same 184 tokens (56 input plus 128 output).

The author compared three configurations — normal responses with pre-estimation off, normal with pre-estimation on, and streaming — across parallelism of 1, 4, and 8, five runs each for 45 main trials. Sequential sends produced a median of 920 tokens, while parallelism of 4 and 8 both produced 1,472. The gap traces to how requests arrive together: four calls are accepted while the running total is still under the cap, and only afterwards does the gateway start returning 403s. The author is careful to say this does not establish a general rule that overshoot scales with parallelism, since at parallelism 8 the first batch alone already hit 1,472. One streaming run out of five also showed 1,472, a difference the author could not trace to an internal cause.

A separate practical point concerns where consumption is measured. In the streaming case the APIM x-lab-consumed header reported 56 tokens, while reading the response to completion showed model usage of 184, because the header arrives before the body finishes. Counting the header as final consumption would drop the 128 output tokens. Across all 72 trials and 795 requests — including 18 control and 9 rate-limit trials — no successful response had missing usage and there were no model-side errors. The stakes here rest on whether the same pattern holds under different conditions, such as long inputs that exceed the remaining budget, or sending the next request as each response finishes rather than waiting for a whole batch. Those are the cases the author intends to test next, and until then the finding is tied to this specific short-prompt, batch-waiting setup.