llm-d: prefix-aware routing cuts p90 TTFT from 92.6 to 0.54 seconds

The llm-d benchmark places routing decisions at the center of LLM serving performance. The test workload simulated 150 customers, each with a 6,000-token shared context and 5 users sending 1,200-token questions, at 3 to 60 QPS. Those shared prefixes needed about 73% of the cluster's KV cache, roughly 6x what one pod could hold, which is the condition under which routing choices matter most.

The gap between routing strategies was wide. Random routing produced a p90 time to first token of 92.6 seconds, load-aware only 94.9 seconds, approximate prefix-aware 31.1 seconds, and precise prefix-aware 0.54 seconds. Load-aware routing did no better than random in this test, because each request landed on a GPU that had to recompute its prefix.

A later llm-d rerun on a newer vLLM, against a plain Kubernetes round-robin Service, showed a smaller gap, but p90 time to first token still dropped by more than 99%. All of this depends on the prefix cache itself working, and a timestamp at the top of a prompt breaks every block after it.