LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

Tao Feng, Fangxu Yu, Haozhen Zhang, Zhongjie Dai, Liangqi Yuan, Zijie Lei, Weizhi Zhang, Kunlun Zhu, Haodong Yue, Keyang Xuan, Ge Liu, Jiaxuan You

A unified infrastructure for developing, evaluating, and deploying LLM routers across diverse tasks and cost constraints.

How can we unify the fragmented landscape of LLM routing methods into a single, evaluatable infrastructure?

Researchers currently struggle to compare LLM routers because each method uses incompatible codebases, distinct formalisms, and isolated evaluation protocols that make it impossible to isolate which design choices actually drive performance. LLMRouter solves this by formulating all routing as a sequential decision process, providing a standardized library where any router is defined by five modular components—context encoders, model encoders, scoring functions, decision rules, and learning signals—that operate on a shared data pipeline. This infrastructure enables direct comparison on xRouteBench, revealing that learned routers consistently outperform fixed-model baselines by selecting smaller, cheaper models for queries that do not require frontier-scale compute.

Paper Primer

The core mechanism hinges on a unified MetaRouter interface that abstracts away the internal logic of diverse routing families. Whether a router uses nearest-neighbor retrieval or autoregressive decoding, the system treats it as a single decision-making agent that maps a state—comprising the query, user context, and interaction history—to a specific model dispatch action.

Learned routing policies provide superior performance-to-cost efficiency compared to fixed-model baselines.

Learned routers identify queries that smaller, cheaper models can solve correctly, avoiding the high cost of always defaulting to the largest available model. 14.6% relative improvement over the strongest fixed-model baseline.

The system automates the construction of routing supervision through a three-stage pipeline: query curation, response collection across a pool of 18 candidate models, and metric scoring. This creates a dense query-model matrix that serves as both the training supervision and the test bed, allowing researchers to swap candidate pools or tasks via simple configuration changes.

Why does the field need a unified formulation for routing?

Existing routers are developed under distinct, incompatible formalisms, making it impossible to determine if performance gains stem from the routing method itself or from differences in the underlying experimental stack.

Does multi-turn routing consistently outperform single-turn routing?

No. The authors find that additional rounds of decomposition and aggregation often introduce redundant information and computational overhead without providing consistent gains over a single, well-chosen routing decision.

The Need for Unified Routing

We expose the fragmented state of LLM routing and motivate a unified decision‑process formulation.

Large language models vary widely in cost and capability, so no single model can serve every query efficiently. Current routing approaches—binary quality predictors, cost‑aware cascades, graph‑based, and agentic routers—are built on disparate formalisms and incompatible codebases, and there is no shared evaluation pipeline to compare them fairly.

Because routing methods are scattered across incompatible architectures, researchers cannot isolate which design choices truly drive performance or reuse components across studies.

**Figure 1.** Overview of LLM routing. Routing is driven by three needs (left), namely cost efficiency, capability matching, and user preference. Our unified formulation (right) casts all of them as one decision process: a context encoder $E_q$ represents the routing state of query, persona, and interaction history, a model encoder $E_m$ represents each candidate, and the router dispatches the query or its sub-queries to selected models and aggregates their responses into the answer. The single-turn, multi-turn, and personalized families differ only in which part of the state they observe.

The field’s current fragmentation of routing approaches hinders reproducibility and fair comparison.

Routing as a Decision Process

We recast all router designs as a step‑by‑step decision process.

Existing routers appear in many incompatible forms—binary selectors, cost‑aware cascades, graph‑based policies, and RL‑trained agents—making systematic evaluation difficult.

Routing is treated as a policy that repeatedly looks at the current query, optional user context, and the answers gathered so far, then either calls another model or stops.

How does this sequential decision process differ from a single‑turn router?

A single‑turn router makes one dispatch and immediately returns a result; the sequential process can call multiple models, updating the history after each call, and uses that history to inform later decisions, enabling adaptive multi‑turn interaction.

Step 1: $h_1 = \varnothing \cup y_1 = \{y_1\}$.

Step 2: New state $s_2=(q,\varnothing,h_1)$ is encoded; $m_1$ now scores higher because $h_1$ matches its expertise, so $a_2=m_1$ is dispatched.

Step 2 response $y_2$ is appended: $h_2 = h_1 \cup y_2$.

Since $T=2$ the terminating action $a_3=K$ is taken, aggregating $y_1$ and $y_2$ into the final answer $y$.

The process can adapt its next choice based on the content of earlier responses, something a one‑shot router cannot do.

**Table.** Comparison of different routing families.

Evaluating Routing Performance

Benchmark design quantifies router quality versus cost across diverse tasks.

GraphRouter achieves the highest average benchmark score across all tracks.

Average score 45.46, beating the second‑best method by 2.1 points.

xRouteBench is a unified benchmark that evaluates LLM routers on diverse scenarios while jointly measuring answer quality and inference cost.

The Performance‑Cost Objective collapses a router’s answer quality and its inference cost into a single scalar, letting users trade accuracy for efficiency.

How does the Performance‑Cost Objective differ from a plain accuracy metric?

Accuracy alone ignores the computational expense of generating an answer. $R$ subtracts a cost term, so a router that achieves the same accuracy but uses fewer tokens receives a higher score, enabling explicit trade‑offs between quality and efficiency.

**Figure 2.** Task composition of xRouteBench. The benchmark covers generic LLM tasks, memory, vision, time-series, and personalized routing, with percentages indicating the proportion of test queries contributed by each dataset.

**Table 1.** Evaluation Protocol. We score each router by a weighted reward $\alpha \cdot \text{perf} - \beta \cdot \text{cost}$. We sweep five weight settings from the quality-only $(\alpha, \beta) = (1.0, 0.0)$ to the heavily cost-weighted $(0.2, 0.8)$. Multi-round and RL-based routers cannot optimize this weighted objective and are therefore run once under a single configuration. On the personalized track, answers are scored by a persona-conditioned LLM judge (DeepSeek-V3.1) as win, tie, or loss (1, 0.5, 0), and the judge's own cost is excluded from the reported cost.

Balancing answer quality against inference cost is essential for realistic LLM routing evaluation.

The LLMRouter Library

LLMRouter Library provides a modular, plug‑and‑play system that unifies routing components into a single API.

The LLMRouter library stitches together data construction, routing logic, training, inference, evaluation, and deployment into a single, interchangeable framework. By exposing only five router components to the user, it turns what used to be a fragmented ecosystem into a plug‑and‑play experience.

Think of the library as a universal socket: any routing strategy can be plugged in, while the surrounding wiring automatically supplies power, handles the plug‑in’s wiring, and routes the output to the device.

Step 1: The matrix is filled with per‑token prices: [[0.5, 0.8, 1.2], [0.5, 0.8, 1.2]].

Step 2: A router (e.g., kNNRouter) encodes each query with $E_q$, each model with $E_m$, computes scores via $g$, and selects the highest‑scoring model $d$ for each row.

Step 3: For Q₁ the router picks M₂, for Q₂ it picks M₁; the resulting actions are stored in the query dictionaries.

Step 4: The Route Engine dispatches Q₁ to M₂ and Q₂ to M₁, collects the responses, and returns the final answers.

This toy example shows that swapping the router class changes only the scoring step; the surrounding Data Engine, Route Engine, and Deployment remain untouched.

How does the LLMRouter library differ from ad‑hoc router scripts that people usually write?

Ad‑hoc scripts embed data loading, scoring, and inference logic together, so changing the routing rule forces a rewrite of the whole pipeline. LLMRouter separates those concerns: the Data Engine, Route Engine, and Evaluation are fixed, and only the four router components ($E_q$, $E_m$, $g$, $d$) need to be reimplemented, making experimentation a one‑line subclass change.

**Figure 3.** Architecture of LLMRouter. The system consists of six modules that support routing data construction, router implementation and training, inference, evaluation, and deployment.

Defining a custom router and its trainer using the LLMRouter library.

Experimental Results

Key results reveal how router performance shifts with cost sensitivity.

The unified LLM Router premise treats routing as a sequential decision process, enabling consistent evaluation. This section now shows how different routers actually perform.

Smallest‑LLM attains the highest average xRouteBench score (80.54) under the performance‑first setting.

Table 2 reports Smallest‑LLM with an average of 80.54, surpassing every other router.

**Figure 5.** Router rankings across the Generic LLM Tasks, memory, vision, and time-series tracks as the cost weight $\beta$ increases. Each cell gives a router's rank under the weighted performance–cost objective, with smaller rank values indicating better performance.

**Figure 6.** Performance–cost trade-offs of routers averaged across the xRouteBench tracks. Each point represents an operating setting with a different cost weight $\beta$, where higher performance and lower per-query inference cost are preferred.

**Table 1.** Comparison of different router methods based on accuracy.

Performance trade‑offs vary markedly with the cost weight $β$, so selecting a router requires matching $β$ to deployment constraints.

Deployment and Multi-Agent Systems

Recall that LLM routing is unified as a sequential decision process, enabling standardized evaluation and deployment.

Routing in deployment moves beyond static benchmarks, exposing routers to live user feedback and multi‑agent coordination.

Real‑user routing uses the OpenClaw server to serve LLMRouter behind Slack, collecting 234 pairwise preference records from fifteen users across 40 sessions.

The table compares the accuracy (Acc.) of various router models.

When the same routers are evaluated on the Slack deployment, GMTRouter, which excelled under persona‑based judging, drops to sixth place, highlighting the gap between simulated and real feedback.

Multi‑agent routing treats each agent node as a separate decision point, selecting the most suitable LLM for its prompt across five coordination topologies.

**Table 4.**

**Figure 7.** Representative multi-agent system architectures and coordination topologies: (a) star-based centralized coordination, (b) hierarchical tree-based delegation, (c) graph-based peer interaction, (d) sequential chain collaboration, and (e) planner–executor–summarizer workflow.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers