ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving

Sangjin Choi, Sukmin Cho, Yifan Xiong, Ziyue Yang, Youngjin Kwon, Peng Cheng

ELDR routes MoE requests by expert locality to shrink per-step weight fetches, reducing decode latency without changing model outputs.

How can we route requests in disaggregated MoE serving to group requests with similar expert usage, thereby reducing the number of distinct experts activated per decode step?

In Mixture-of-Experts (MoE) serving, decode latency is governed by the union of distinct experts activated by a batch. Standard load-balancing routers ignore this, scattering requests across workers and forcing each to load a wide, fragmented set of expert weights from memory. ELDR introduces expert-locality-aware routing: it uses prefill-time expert activations to build a signature for each request, then routes requests with similar signatures to the same decode worker. This co-location shrinks the per-step active-expert union, reducing memory-bandwidth pressure. Across three MoE models and two workloads, ELDR reduces median Time-per-Output-Token (TPOT) by 5.9–13.9% over the strongest load-balancing baselines, with no change to model outputs.

Paper Primer

The core challenge is that MoE decode is memory-bandwidth bound, not compute-bound. Because each decode step must fetch the weights for every expert selected by any token in the batch, a router that mixes unrelated requests forces the worker to load a larger, more diverse set of experts, spiking latency.

ELDR solves this by treating expert locality as a first-class routing axis: it captures a request's "expert signature" during prefill, clusters these signatures offline to define locality regions, and uses a locality-band routing rule online to balance expert affinity against instantaneous worker load.

ELDR significantly reduces median TPOT across diverse MoE models and workloads.

Evaluations on Qwen3-30B-A3B, GPT-OSS-120B, and Gemma-4-26B-A4B show consistent improvements over load-balancing baselines like Join-Shortest-Queue (JSQ) and Power-of-Two-Choices (P2C). 7.0–13.9% reduction in median TPOT for task workloads; 5.9–10.0% for language workloads.

To maintain performance with prefix caching, ELDR uses a block-granular signature cache co-indexed with the KV cache. This allows the router to reconstruct the full expert signature for partial cache hits without re-running the gating networks, ensuring routing remains coherent even as KV blocks are evicted or reused.

Why is expert locality visible at prefill if the decode-time expert choices are not yet known?

The model's gating networks are consistent; prefill and decode activations correlate strongly (0.70 to 0.92). The prefill phase processes the prompt through the same gates that will route decode, exposing a request-specific expert footprint that predicts future usage.

Does ELDR sacrifice load balancing to achieve expert locality?

No. ELDR uses "locality-band routing," which restricts routing to a subset of workers that match the request's expert signature, then selects the least-loaded worker within that band. This ensures that load-balancing remains the tie-breaker for instantaneous queue variance.

ELDR is a lossless optimization: it changes only which worker serves a request, never the gating decisions themselves, ensuring that model outputs remain identical to standard top-k gating.

Introduction

Decode latency in PD‑disaggregated MoE serving hinges on expert overlap, so we propose ELDR to route by locality.

In PD‑disaggregated serving each request is assigned a decode worker after its prompt has been prefixed. Existing decode routers balance only the per‑worker load, assuming that equal load yields equal latency. For mixture‑of‑experts (MoE) models this assumption fails because latency is set by the union of distinct experts activated by the batch at each decode step.

Prefill and decode are run on separate pools of workers, letting the system allocate resources independently to the compute‑heavy prompt phase and the latency‑sensitive token‑generation phase.

Expert locality provides a second routing axis: requests that activate overlapping experts share more of the same weights during generation, reducing the per‑step union of distinct experts and thus latency. Empirically, same‑domain decode batches activate 17–21 % fewer distinct experts per step than mixed batches, making locality a sizable, structured latency knob.

ELDR exploits this by turning each request’s prefill activations into an expert signature that predicts which experts will be needed during decode. Offline, balanced K‑means clusters the signature space, assigning each cluster to a decode worker. Online, a locality‑band selects the workers whose centroids lie within a similarity band of the request’s signature, then routes to the least‑loaded among them. A block‑granular signature cache, co‑indexed with the KV cache, keeps signatures exact even when prefix caching yields partial prefill information.

The shift from load‑only routing to locality‑aware routing is the key to reducing decode latency for MoE models.

The image displays the author list and affiliation details for a research paper, followed by the beginning of the abstract text.

Why Expert Locality Matters

Decode latency hinges on how many distinct experts fire per step.

In a Mixture‑of‑Experts (MoE) model, each decode step must load the weights of every expert that any token in the batch contacts. When the batch is split across many experts, weight reuse collapses and the memory‑bandwidth bottleneck dominates latency.

An “activation signature” is the per‑expert usage pattern a request leaves behind after its prompt has been processed.

**Figure 1.** Decode-phase per-expert activation relative to the cross-domain mean, for three MoE models along task (top) and language (bottom, WildChat [37]) domains at each model’s most discriminative layer. Darker is above-average (below-average clipped to white); experts are reordered per panel into contiguous per-domain blocks. Each domain over-activates a distinct subset of experts. Task domains: Code [1, 4, 17, 40], Math [5, 11, 13, 19, 20], Medical [12, 14, 15], and Legal [3].

**Figure 2.** MoE layer latency scales with active experts, not batch size (single MoE layer, one MI300X).

**Figure 3.** Prefill expert activation predicts decode activation. Each point is one expert (normalized prefill x vs. decode y, pooled over domains); points near the diagonal are experts used about equally in both phases.

**Figure 4.** Same-domain batches (blue) activate fewer experts per decode step than mixed-domain (orange), for task (top) and language (bottom) across three MoE models.

Because the prefill signature is observable before any output token is generated, a routing policy can place a request onto a decode worker that already hosts requests with overlapping signatures, thereby shrinking the active‑expert set and cutting latency.

The ELDR Routing Mechanism

ELDR routes decode requests by expert‑signature similarity to cut active‑expert count.

In PD‑disaggregated MoE serving, latency is driven not only by how many requests sit on a worker but by how many distinct experts those requests activate per decode step. Existing routers balance load but ignore that colocating expert‑similar requests can dramatically shrink the active‑expert set.

WildChat traffic is heavily skewed: English and Chinese together account for roughly 75 % of requests (see Figure 6), so a locality‑aware policy can concentrate the bulk of work on a few workers.

ELDR assigns each decode request to the worker whose current expert set most overlaps with the request’s predicted expert usage, as encoded in its expert signature.

How does ELDR differ from a plain load‑balancing (e.g., round‑robin) router?

Round‑robin ignores the expert activation patterns and may scatter requests that would share experts across many workers, inflating the per‑step active‑expert count. ELDR explicitly measures predicted overlap via signatures and prefers workers where that overlap is high, while still respecting a hard load ceiling.

The signature cache stores the expert signature for each KV‑cache block so that later requests reusing that block inherit its expert footprint without recomputing it.

Why can’t we simply cache the whole‑request signature instead of per‑block signatures?

A whole‑request signature only describes a full prompt. When a request reuses only a prefix, the missing suffix would be omitted, causing the router to underestimate its expert usage and misroute it. Per‑block signatures allow arbitrary prefix reuse while preserving the correct aggregate footprint.

Normalize r₁ → s₁ = [0.89, 0, 0.45, 0].

Normalize r₂ → s₂ = [0.71, 0.71, 0, 0].

Normalize r₃ → s₃ = [0.71, 0, 0.35, 0.71].

Compute pairwise cosine similarities: sim(s₁,s₂)=0.45, sim(s₁,s₃)=0.78, sim(s₂,s₃)=0.50.

Average similarity $\rho$ = (0.45 + 0.78 + 0.50)/3 ≈ 0.58, indicating moderate cohesion.

Higher average similarity means the three requests will likely share many experts during decode, so ELDR will route them to the same worker.

Client sends request metadata to the ELDR router.

Router performs prefix‑aware routing (step 1) to a prefill worker.

Prefill worker computes per‑block expert signatures and stores them in the signature cache (step 2).

When prefill finishes, the router receives the aggregated request signature (step 3).

Router executes locality‑band routing (step 4) by comparing the request signature to each worker’s centroid and selects the best‑matching decode worker.

KV states are transferred from the prefill worker to the chosen decode worker (step 5).

ELDR routing decision – selects decode worker based on signature similarity and load cap.

**Figure 6.** ELDR architecture: offline fitting of one centroid per decode worker over expert signatures, then online routing at the prefill-decode handoff by signature similarity, subject to load.

**Figure 5.** WildChat [37] request volume is heavily skewed: English and Chinese alone are ~75% of requests.

**Figure 10.** ELDR stores expert signatures at KV cache block granularity: the signature cache is co-indexed with KV cache.

Evaluation and Results

ELDR cuts decode latency and active‑expert load across diverse MoE models.

ELDR reduces median TPOT by up to 30 % across three MoE models compared with round‑robin routing.

Figures 11 and 12 show ELDR (red line) consistently achieving the lowest median TPOT across request rates for Qwen3‑30B‑A3B, GPT‑OSS‑120B, and Gemma‑4‑26B‑A4B.

vLLM is a high‑throughput inference engine that batches prefill and decode steps across many requests while keeping a shared KV cache.

How does ELDR’s signature differ from a naïve count of expert activations?

ELDR first reweights raw counts with IDF to suppress ubiquitous generalist experts, then discards non‑informative layers via a learned mask. A naïve count would treat every activation equally, drowning out the rare specialists that actually separate requests.

**Figure 7.** Signature quality $\rho$ (Eq. 1) for six candidate transformations $T$. Bars are the mean across six cells (3 models $\times$ 2 workloads); whiskers span the per-cell min/max.

**Figure 8.** Cumulative $\rho$ (Eq. 1) versus the number of layers kept under greedy layer selection. One panel per model; task (blue) and language (orange) shown separately. The star marks the peak $N^*$ chosen by ELDR’s offline fit.

**Figure 9.** PCA of calibration signatures with Hungarian-balanced centroids (K=8 for legibility), colored by task domain (top) or top-4 WildChat [37] language (bottom).

**Figure 11.** TPOT (median, p99) and median TTFT vs request rate on the task workload at 8P16D.

**Figure 12.** TPOT (median, p99) and median TTFT vs request rate on the language workload at 8P16D.

**Figure 16.** Mean %$\Delta$ vs RR (five rates, 20–100 qps; 8P16D) for six cells (three models $\times$ two datasets) at four $\tau$ values.

ELDR consistently reduces TPOT across diverse MoE models.

Ablations and Sensitivity Analysis

ELDR’s ablations show each design choice measurably improves latency and expert locality.

We evaluate ELDR on a 5‑node AMD MI300X cluster using four open‑source MoE models and two heterogeneous workloads.

ELDR reduces median TPOT by up to $13.9\%$ across all models and workloads.

Mean improvement over five request rates; ELDR beats every baseline at every rate.

On the task workload ELDR improves median TPOT by $7.0\!-\!13.9\%$ and tail TPOT by $3.4\!-\!6.0\%$ over the best load balancer.

Results reported in §6.1 for code, math, medical, and legal domains.

On the language workload ELDR improves median TPOT by $5.9\!-\!10.0\%$ over the best load balancer.

Aggregated across Qwen3‑30B‑A3B, GPT‑OSS‑120B, and Gemma‑4‑26B‑A4B.

ELDR adds $0.86\,$ms (1.2 % of median TTFT) per request at serving time.

Table 1 reports the per‑request overhead broken down by component.

Active‑expert reduction: ELDR cuts the per‑step distinct‑expert count by $22\%$ versus Round‑Robin.

Measured across decode batch sizes in §6.3.

Signature choice (count·idf) yields an extra $3$ pp improvement in TPOT P50 on average.

Across six cells the IDF‑reweighted count outperforms raw counts.

Hungarian‑balanced K‑means improves median TPOT by up to $12.6\%$ and tail TPOT by up to $6.8\%$ over vanilla K‑means.

Results in §6.3 for $\tau$ = 0.1.

Locality band $\tau$ = 0.1 reduces median TPOT by $5.2\%$ relative to Round‑Robin and eliminates tail regressions.

$\tau$ sweep results in §6.3.

Prefix‑cache composition preserves ELDR’s $13\%$ median TPOT advantage over RR when the cache is on.

Figure 17 shows identical TPOT gaps with cache off and on.

Scaling decoder pool size yields larger gains: 8P24D achieves a $10.2\%$ median TPOT reduction versus RR.

Figure 16 reports mean reductions for 8P8D (8.0 %), 8P16D (9.8 %), 8P24D (10.2 %).

On the 235B model ELDR cuts median TPOT by $4.3\%$ and tail TPOT by $2.0\%$ across all request rates.

Figure 19 shows consistent improvements over RR.

**Table 1.** ELDR runtime overhead (Qwen3-30B-A3B, task, 8P16D, 60 req/s; median TTFT 69 ms).

**Figure 14.** TPOT P50/P99 % $\Delta$ vs. RR at $r=60$ req/s (8P16D, six (model, dataset) cells) for two signature transforms: the IDF-reweighted top-$k$ count (count-idf) vs. the continuous softmax gate (gate-prob). Rest of the recipe fixed (greedy mask, balanced $K$-means $K=16$, $\tau=0.1$).

**Figure 15.** Mean %$\Delta$ vs. RR over five request rates (20–100 qps) at 8P16D with $\tau=0.1$.

**Figure 17.** Composition with prefix caching on GPT-OSS-120B in the task domain on an 8P16D cluster at r=100 req/s, with requests sampled cyclically from 2000 task prompts.

**Figure 18.** TPOT (median, p99) for Qwen3-235B-A22B on language workload at 2P8D with TP=4 and EP=4 per instance (40 GPUs across 5 nodes).

Background

Provides concise context on related work and foundational concepts.

This section surveys prior approaches to MoE serving and situates our contributions within the broader literature.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers