New batch-level routing framework optimizes LLM query distribution while managing costs, GPU resources, and capacity constraints more effectively than per-query methods.

Researchers developed a resource-aware routing system that assigns multiple queries to LLMs as batches rather than individually, improving control over total costs and resource usage

A robust variant accounts for uncertainty in LLM performance predictions, improving accuracy by 1-14% compared to non-robust approaches depending on the performance estimator used

Batch-level routing outperforms traditional per-query methods by up to 24% when dealing with adversarial or non-uniform batching scenarios

An offline instance allocation procedure balances quality and throughput across multiple models, with experiments validated on two multi-task LLM benchmarks