Tsinghua's TokenRouter switches small and big LLMs token by token

TokenRouter arrives as an answer to a practical limit. Request-level routing, where a decision model assigns each incoming question to one model at the start, became common thanks to tools like RouteLLM and OpenAI's Decisions API, but it commits the whole answer to one model even when only a small part is genuinely hard. Token-level routing — deciding per token — was already proposed by methods such as R2R, CITER, Co-LLM and R-Stitch, but existing servers assume a single model handles each request end to end, so switching models mid-answer wasted time on step misalignment, batch admission waits and cache management.

The paper's framing is "request-centric authoring, model-centric execution." TokenRouter keeps a sub-server per model, adds a third loop for passing requests between them, and avoids shipping KV caches across models, instead sending only the tokens the other model has not seen. The paper also adds delayed batching so the large model can accumulate requests before processing, with a math model that predicts the optimal threshold B per concurrency level.

An independent Mac reproduction of the R2R flow with Qwen3-0.6B and Qwen3-32B, on 30 MATH-500 problems, found 23 correct versus 20 for the small model alone and 27 for the large model alone, with the large model writing 28.3% of tokens and the run taking about 56% of the large-model-only time. Two findings stand out: the tokens routed to the large model were reasoning-direction words like "since," "But" and "Wait," not numbers; and the speedup collapsed on a single Mac versus a decision-model request router.