TokenRouter: Efficient Serving System for Token-Level LLM Routing
Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong, Yi Ge, Yichen You, Yu Wang
TokenRouter decouples token-level routing logic from serving infrastructure to eliminate synchronization bottlenecks.
How can we build a serving system that efficiently handles token-level LLM routing without the performance bottlenecks of existing request-level serving architectures?
Token-level routing improves LLM efficiency by dynamically switching models mid-response, but existing serving systems force these models to synchronize at every step, causing massive idle time and batch fragmentation. TokenRouter solves this by treating each LLM as an autonomous subserver that communicates via an asynchronous route-send-receive interface, allowing models to progress at their own pace. This architecture achieves 2.01–64.15× higher decoding throughput than standard single-LLM serving systems across diverse routing algorithms and workloads.
Paper Primer
Existing serving systems are built on the assumption that a single model generates the entire response, forcing all requests in a batch to wait for the slowest model at every step. Token-level routing breaks this by requiring frequent model switches, which current systems handle by stalling the entire pipeline, leading to severe step desynchronization and inefficient batch admission.
TokenRouter implements a request-centric programming model where developers define routing logic as a simple flow, while the runtime executes it using model-centric subservers. The system uses a decoupled tri-loop execution to handle inter-model handoffs asynchronously, ensuring that one model's latency does not block the entire system.
TokenRouter significantly boosts decoding throughput compared to existing serving implementations.
Across 15 algorithm-workload combinations, TokenRouter consistently outperformed baselines.
The system maintains high throughput even as output length increases.
When comparing low-effort (2,048 tokens) to high-effort (8,192 tokens) reasoning, TokenRouter's throughput remained steady, whereas standard serving baselines lost 58.1–85.2% of their throughput.
Why is this system necessary if existing frameworks like vLLM or SGLang already support batching?
Existing systems are designed for single-model serving where requests stay with one model for the duration of the response. Token-level routing requires data-dependent model switches at every token, which causes irregular arrivals and fragmented batches that standard synchronous schedulers cannot handle efficiently.
What is the core trade-off in the delayed-batching scheduler?
The scheduler buffers incoming requests for a short, controlled interval to form larger batches. This trades a small increase in initial waiting time for significantly higher overall throughput by reducing the frequency of fragmented, small-batch executions.
TokenRouter transforms token-level routing from a theoretical efficiency gain into a practical serving framework, allowing developers to implement complex routing policies without manually managing low-level synchronization or batching logic.
Introduction to Token-Level Routing
Serving token‑level routed LLMs suffers from synchronization and batching bottlenecks.
Token‑level routing promises large efficiency and quality gains, yet existing serving stacks were built for single‑LLM workloads. The mismatch creates three concrete pain points: step desynchronization, batch admission delays, and a steep implementation learning curve.
Each generated token may be processed by a different LLM, allowing the system to pick the cheapest model that can handle the token’s difficulty.
Routing decisions are made once per request (session or query) and the chosen LLM processes the entire response.
**Figure 1.** TokenRouter improves serving efficiency for token-level routing. (a) At concurrency 8, engineering optimizations yield a 1.71x throughput gain, and TokenRouter achieves 2.76x overall, relative to the official R2R implementation. (b) Throughput–speed trade-off. TokenRouter achieves 18.58x higher throughput than R2R under a stricter SLO.
**Figure (a)** Routing levels in LLM serving.
The shift from request‑level to token‑level routing requires a fundamental change in serving system architecture.
Programming Interface
Defines the route‑send‑receive API that lets developers program token‑level routing.
TokenRouter exposes a tiny programming surface: developers implement three callbacks—route, send, and receive—while the runtime orchestrates asynchronous model‑centric execution.
Think of a request as a token‑by‑token traveler; at each step the traveler asks “where next?” (route), hands over its luggage (send), and the new host unpacks it (receive).
The request lifecycle can be written as a chain: receive → decode → route → send → peer receive → peer decode → …, which the three callbacks capture in a request‑centric programming model.
**Table 1.** Routing behavior of the evaluated token-level routing algorithms. Each entry under Model $i$ or Model $j$ specifies the condition under which that model routes generation to another model.
System Design and Scheduling
TokenRouter’s runtime decouples routing from model execution to cut synchronization overhead.
The TokenRouter runtime builds on a multi‑subserver layout, exposing a single external API while each subserver runs its own LLM and scheduler.
**Figure b.** TokenRouter system architecture.
Instead of a single decode loop that stalls when any model lags, TokenRouter runs three independent loops—client‑server, decoding, and an inter‑model loop—so routed tokens can hop between models without pausing the local batch.
Why does TokenRouter need a third loop instead of reusing the existing decoding loop?
Reusing the decoding loop would require all models to stay in lockstep; the third loop isolates peer‑to‑peer traffic, letting each subserver advance at its own pace and eliminating the global barrier that causes idle time.
When a token is handed off to another model, TokenRouter marks the request as “pending” so the scheduler skips it but preserves its KV‑cache and prefix, avoiding costly recomputation on return.
How does the pending state differ from simply dropping and re‑adding a request?
Dropping discards the existing KV‑cache, forcing a fresh prefix match on resumption. The pending state retains the cache, so the resumed request picks up exactly where it left off, saving both compute and memory.
In the naive eager‑async approach, each incoming token is immediately forwarded to the LLM peer, causing large batch admission delays because the peer must wait for enough same‑model tokens to form a batch.
Why isn’t the eager‑async policy optimal for token‑level routing?
Because routing often produces isolated tokens for the LLM, eager‑async forces the LLM to run many tiny batches, which dramatically raises the average latency compared to batching similar‑model tokens together.
All subservers synchronize at step barriers; the next decoding step starts only after every model finishes the current step.
When would Step‑Sync be preferable despite its synchronization cost?
When models have very similar per‑step latency and the workload demands strict ordering, Step‑Sync guarantees deterministic timing at the expense of occasional idle time.
Only one model runs at a time; requests for inactive models stall until the active model finishes its batch.
How does Model‑Sync differ from the naive single‑model serving loop?
Model‑Sync still keeps a single active model but adds explicit barriers between models, whereas a naive loop would simply process one model continuously without ever switching, preventing any routing at all.
Instead of launching a batch the instant a token arrives, the scheduler buffers requests and only starts a batch when the buffer reaches a threshold $B$, allowing more same‑model tokens to be processed together.
At t=0, the buffer holds 1 token (req 1) → size $< B$, so no batch is launched.
At t=1, the buffer holds 2 tokens (req 1, req 2) → size $= B$, the scheduler launches a batch of size 2.
The LLM processes the batch, producing outputs for req 1 and req 2.
At t=2, req 3 arrives; the buffer resets to 1 token → waits for another token before launching the next batch.
Delaying batch launch by one time unit lets the LLM handle two tokens at once, cutting per‑token compute by roughly half while only adding a modest latency for the first request.
What happens if the threshold $B$ is set too high?
Requests may wait excessively long, starving the LLM of work and reducing overall throughput; the analytical model in Appendix D quantifies this trade‑off and yields the optimal $B^*$.
**Figure 4.** Scheduling strategies of token-level routing. (a) Step-Sync sets step barriers; the next step starts only after all models finish the current one. (b) Model-Sync runs one model at a time; the inactive model's requests stall. (c) Eager-Async processes peer requests immediately, causing large batch admission delay. (d) TokenRouter preserves asynchronous scheduling but delays LLM requests until enough same-model requests accumulate, amortizing the cost across a wider batch.
Table 1 compares routing decisions across several token‑level routing algorithms, showing that TokenRouter’s high‑entropy token handoff aligns with the “high‑entropy token” condition used by R‑Stitch.
Experimental Evaluation
TokenRouter delivers large throughput and latency gains across all token‑level routing baselines.
TokenRouter improves throughput up to 64.15× over the strongest baseline.
Figure 5 shows TokenRouter’s throughput dominating the Official Code and Std. Serving bars across all methods.
Throughput scales 8.61× when concurrency increases from 1 to 16.
Figure 5 reports the throughput curve for TokenRouter across concurrency levels.
TokenRouter achieves up to 3.21× higher throughput than R2R across model pairs.
Table 3 lists gains ranging from 1.99× to 3.21× for four SLM‑LLM pairings.
CITER routes tokens by consulting a small‑model router that decides, with a high confidence threshold ($\tau$ = 0.98), whether to keep the token on the small model or hand it to the large model.
How does CITER differ from a naïve “always use the small model unless the large model is needed” rule?
CITER learns a data‑driven confidence score per token, rather than using a fixed heuristic; the learned threshold $\tau$ = 0.98 lets it adapt to token difficulty, avoiding unnecessary large‑model calls while preserving quality.
R2R (Router‑to‑Router) passes each token through a dedicated router that decides, based on a fixed threshold, whether the token should be processed by the small or the large model.
Why does R2R suffer higher latency than TokenRouter despite using the same routing idea?
R2R processes each token synchronously; after a token is routed, the serving system must wait for the full request to finish before issuing the next token, incurring extra prefix‑matching and KV‑cache updates that TokenRouter’s handoff‑and‑resume design eliminates.
Setup details: all experiments run on an 8 × A100‑80G server; two‑model algorithms split the small model onto one GPU and the large model onto two GPUs with tensor parallelism. Baselines include the released Official Code implementation and a custom Std. Serving stack built on SGLang.
**Figure 5.** Throughput (token/s) comparison across serving systems at concurrency 4.
**Figure 6.** Latency (s) comparison across serving systems at concurrency 4.
**Figure 7.** Throughput–accuracy trade-off. QR denotes query-level routing from RouteLLM.
Tables 2, 3, 4, 6, and 10 provide the quantitative backing: Table 2 reports TTFT, throughput, and latency for each algorithm; Table 3 quantifies the 1.99–3.21× gains over R2R; Table 4 lists the model‑pair and benchmark configurations; Table 6 details the ablation of CUDA‑graph and async optimizations; Table 10 breaks down per‑step latency for Std. Serving.
TokenRouter consistently outperforms existing serving systems across all evaluated token‑level routing algorithms.
Ablation Studies
Ablation studies quantify how each TokenRouter component affects throughput and latency.
We evaluate the contribution of each TokenRouter subsystem by disabling it in isolation and measuring the resulting decoding speed.
TokenRouter delivers 2.01–64.15× higher decoding throughput than existing serving stacks.
Across routing algorithms, workloads, and model pairs, the full system consistently outperforms the baselines.
**Figure 8.** Throughput (token/s) comparison across serving systems at concurrency 1.
**Figure 9.** Latency (s) comparison across serving systems at concurrency 1.
Table 5 isolates the effect of each subsystem. Removing delayed batching, async execution, the Router CUDA graph, the LLM CUDA graph, or the SLM CUDA graph (≈ R2R) each lowers the reported metric for $N=8$, $N=4$, and $N=1$.
Throughput Modeling Theory
We formalize token routing as a stochastic throughput model that predicts steady‑state token processing.
Token‑level routing aims to improve inference efficiency, but synchronization overhead can dominate runtime. This section builds the probabilistic model that captures routing dynamics and yields a closed‑form throughput estimate.
The model treats each routing decision as a draw from a multinomial distribution, then aggregates expected token commits to compute long‑run throughput.
Model 1 draws $\mathbf{m}_1\sim\text{Multinomial}(3,\{0.6,0.4\})$, e.g. $m_{10}=2$, $m_{11}=1$.
Model 2 draws $\mathbf{m}_2\sim\text{Multinomial}(2,\{0.3,0.7\})$, e.g. $m_{20}=0$, $m_{21}=2$.
Compute $g(s,M)=c_{10}m_{10}+c_{11}m_{11}+c_{20}m_{20}+c_{21}m_{21}=2+1+0+2=5$ committed tokens.
Assume the step duration $\tau(s)=2$ (two decoding sub‑steps). The contribution to throughput for this transition is $5/2=2.5$ tokens per sub‑step.
Even with small batches, the expected throughput is driven by the routing probabilities $p_{ij}$; skewed probabilities can dramatically lower $E[g(s,M)]$ despite identical batch sizes.
How does this throughput model differ from simply measuring average tokens per second on a running system?
The model predicts the *expected* token commit rate from the probabilistic routing policy and state dynamics, not from observed runtime measurements that conflate scheduling delays, hardware variability, and transient load spikes.
Throughput Model Derivation
Derives the throughput model and optimizes delayed‑batching thresholds to maximize token‑level routing efficiency.
Having built the throughput model, we now turn to the concrete problem of choosing the delayed‑batching thresholds B that maximize that throughput while respecting system constraints.
To keep the system alive we must ensure that no model ever waits for an empty batch. This yields the feasibility condition (1/M)($B_i$‑1) < N for each model index i, which defines the finite feasible set ℬ_M.
Instead of fixing a single batch size for all models, we let each model wait until its own threshold $B_i$ is reached, then launch a batch—this adapts to heterogeneous request loads and keeps every model busy.
State $s_1$: model 1 has 1 pending request, model 2 has 0. $\pi_B(s_1)=0.4$ (computed from the Markov chain).
State $s_2$: model 1 has 2 pending requests (reaches its threshold), model 2 has 1. $\pi_B(s_2)=0.6$.
Compute numerator: $\pi_B(s_1) b_{11} p_{11} + \pi_B(s_2) b_{21} p_{21} = 0.4\cdot1\cdot0.9 + 0.6\cdot1\cdot0.85 = 0.36 + 0.51 = 0.87$.
Compute denominator: $\pi_B(s_1)\tau(s_1) + \pi_B(s_2)\tau(s_2) = 0.4\cdot1.5 + 0.6\cdot1.0 = 0.6 + 0.6 = 1.2$.
Throughput $T(B)=0.87 / 1.2 \approx 0.73$ requests per time unit.
Even with a tiny system, adjusting thresholds changes the stationary distribution enough to shift throughput noticeably; the optimizer exploits this sensitivity.
How does this threshold optimization differ from simply fixing a global batch size for all models?
A global batch size forces every model to wait for the same number of requests, which can leave fast‑launching models idle while slower models catch up. By assigning individual thresholds $B_i$, each model launches as soon as its own queue reaches $B_i$, preserving high utilization and allowing the optimizer to balance cost and latency per model.
Empirical Model Validation
We validate the throughput model on R2R routing, confirming its predictions across concurrency and batching.
We empirically validate the throughput model on the representative token‑level routing algorithm R2R [5] with $M=2$ models (model 0 = SLM, model 1 = LLM). The routing‑probability matrix $P$ and token‑commit matrix $C$ follow the paper’s indexing convention, encoding the probability of sending a token to the LLM and the commitment of tokens to models, respectively.
**Figure 10.** Predicted and measured R2R throughput under different concurrency levels and delayed-batching thresholds. We fix the SLM threshold at $B_0 = 1$ and vary the LLM threshold $B_1$. An asterisk (*) marks the highest throughput for each concurrency level in each panel.
These results confirm that the throughput model accurately reflects the effect of delayed batching, and that the analytically optimal thresholds it predicts are realized in practice, providing a reliable guide for selecting throughput‑optimal batching parameters.
Supplementary Performance Data
Key throughput and latency numbers from the supplementary tables are presented.
The following tables summarize throughput and latency measurements for the routing strategies evaluated in the paper.
The table presents a comparison of different routing algorithms (R2R, R-Stitch, Co-LLM, and CITER) across two deployment setups (Overlapping and Non-overlapping) for two different values of N (1 and 4).
**Table 8.** Throughput (token/s) comparison across routing strategies under different concurrency and overlap settings.
The table displays the latency and ratio for various operations, including Prefix matching, Locking cache nodes, SLM inference, Releasing locks, Update radix cache, and Others.
Beyond the core mechanisms, TokenRouter introduces several optimizations that address assumptions broken by token‑level routing.
Token‑level routing forces multiple LLMs to share a GPU’s KV‑cache, so TokenRouter computes a joint allocation that fits within a shared budget.
Since token‑level routing makes extend batches the dominant path, TokenRouter captures CUDA graphs over a range of extend batch sizes to avoid per‑step kernel launches.
Routing decisions now occur after every forward pass, so moving the router onto the same GPU as the LLM eliminates costly cross‑process IPC.
All sub‑servers need a consistent view of the active request count to apply the delayed‑batching threshold correctly.
Implementation Details
Implementation specifics of TokenRouter and the delayed‑batching model.
TokenRouter is built on SGLang 0.5.1. The server interface extends SGLang’s FastAPI front‑end to accept multiple model ports, and subservers exchange messages via local ZeroMQ IPC.
Each subserver runs the three‑stage routing loop in a dedicated subprocess event loop; coordination between subservers occurs only through the peer‑message channel described in Section 4.2. All routing algorithms from Section 5 are instantiated as scheduler subclasses that implement the three core functions from Section 3.
**Table 11.** Key notations used throughout the analysis.
We now formulate a mathematical model to optimize the delayed‑batching threshold introduced in Section 4.3. The model expresses system throughput as a function of the threshold vector $B$ and serves as the basis for the optimization in Sections D.2 and D.3.
We assume routing probabilities are stationary across decoding steps, so $p_{ij}$ does not change over time. Under this assumption, if $p_{ii}$<1 the number of consecutive steps a request stays on model i follows a geometric distribution with success probability 1‑$p_{ii}$. Routing decisions are taken independently for each request.
For any state s, the holding time $\tau$(s) is the minimum residual time among models with an active batch; the models whose batches finish at $\tau$(s) trigger the next state transition. Starting from an initial state, the set of states reachable with positive probability under the routing and scheduling dynamics is denoted $R_B$, which is finite and can be enumerated by breadth‑first search.