FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica

FreeToken enables frontier-scale MoE inference on consumer hardware by dynamically balancing expert execution between GPU cache and CPU compute.

How can we efficiently serve Mixture-of-Experts (MoE) models on edge devices when the total expert pool size exceeds the available GPU VRAM?

Frontier Mixture-of-Experts (MoE) models are too large for consumer GPU memory, forcing existing edge systems to rely on static expert placement that leaves hardware resources underutilized and struggles with the dynamic routing of agentic workloads. FreeToken treats the entire machine as a unified platform, using bandwidth-adaptive execution to split expert misses between PCIe transfer and direct CPU execution based on real-time hardware measurements. On an RTX 5090, FreeToken achieves 1.5–2.3× higher decode throughput than state-of-the-art baselines and maintains stable performance across complex agentic tool-calling sessions.

Paper Primer

FreeToken manages the expert pool as a two-level hierarchy: a complete set of experts resides in host memory, while a shared, elastic cache on the GPU tracks the most frequently routed experts. The system coordinates this hierarchy using a $q^\star$ policy that dynamically partitions missing experts into a cache-fill set (transferred to GPU) and a CPU-execution set (processed in-place), ensuring that neither the PCIe link nor the CPU bandwidth remains idle.

FreeToken significantly improves decode throughput on consumer hardware.

Across five consumer systems, FreeToken achieves 1.3–2.1× higher decode throughput compared to existing edge serving engines like llama.cpp and KTransformers.

The system maintains performance stability during agentic workloads.

Decode rates remain within 12% of single-turn performance during multi-turn agentic tool-calling, whereas competing systems degrade substantially as context grows. Worst-case time-to-first-token (TTFT) stays below 44s, preventing timeouts common in baseline systems.

Why does FreeToken outperform systems that simply pin "hot" experts to the GPU?

Static pinning fails because routing shifts with every token and workload phase. FreeToken uses a shared LRU cache that follows the router's current working set, and it offloads residual misses to the CPU rather than forcing them through a saturated PCIe link.

How does FreeToken handle the resource variability of personal machines?

It treats GPU memory as an elastic resource that can be resized at runtime without restarting the engine. By measuring host-memory and PCIe bandwidth at deployment, it dynamically adjusts the split of work between the CPU and GPU to match the specific machine's capabilities.

Introduction and Overview

FreeToken unifies edge GPUs and CPUs to serve large MoE models interactively on consumer hardware.

Recent open‑weight models such as Kimi‑K3, GLM‑5.2, and DeepSeek‑V4‑Flash are closing the capability gap with proprietary systems, yet the cost of running them remains prohibitive for most users. Hosting these models still requires datacenter‑class GPU clusters, and even when offered via APIs the sustained inference expense is high for individuals and small teams.

Millions of consumer machines already contain discrete GPUs, but the missing piece is a serving system that can treat each heterogeneous device as a unified inference platform. MoE layers contain hundreds of experts while routing each token through only a few, so the active parameter footprint fits within a modern GPU’s memory, yet the full expert pool often exceeds VRAM, forcing inactive experts into host memory.

Existing edge‑serving systems such as llama.cpp, KTransformers, and Ollama address only fragments of the problem. During prefill, long prompts cause the working‑set sparsity to disappear, requiring repeated streaming of experts; during decode, sparse token activation still incurs cache misses that static placement cannot handle; and the wide variability of GPU capacity, PCIe bandwidth, and CPU resources defeats any single static scheduling policy.

FreeToken tackles these challenges with three coordinated ideas. First, bandwidth‑adaptive execution double‑buffers expert movement with computation during prefill and partitions cache‑miss handling between GPU and CPU during decode using a q⋆ policy. Second, semantic‑aware caching anchors recurrent‑state checkpoints at semantic boundaries and employs a shared LRU expert cache to exploit inter‑token routing locality. Third, elastic edge resource management dynamically resizes the GPU expert cache at safe points, rebuilding it without restarting the engine.

**Figure 1.** FreeToken serves the models on the cost–capability Pareto frontier, at interactive speed on consumer hardware. (a) Blended API list price (9:1 input:output mix, following the token economics measured on real coding-agent traces (Zhu et al., 2026)) versus Code Arena Elo (LMArena, 2026) for representative hosted models. Blue squares mark models FreeToken serves, tagged with the consumer GPU class that serves them; the frontier segment from DeepSeek-V4-Flash to GLM-5.2 is exactly this set. Kimi-K3 releases open weights but exceeds consumer memory (594 GB); Qwen3.5-35B stands in for its successor Qwen3.6-35B, which has no arena rating yet. (b) Mean decode throughput on real agentic workloads for the strongest model each hardware tier holds (coding agents on the first two tiers, a math agent on the third), against actively maintained edge engines. The dashed line marks the median decode speed of Codex in production traces (33 tok/s (Zhu et al., 2026)); × marks configurations an engine cannot serve.

FreeToken enables interactive MoE serving on edge hardware.

The Edge MoE Bottleneck

Edge MoE serving is limited by transfer overhead, recomputation, and volatile resources.

Edge devices cannot keep the full expert pool in GPU memory, so each token activates only a tiny fraction of parameters, creating a severe mismatch between model size and available fast memory.

Prefill incurs large transfer overhead because the entire expert pool must be streamed for the thousands of tokens processed per layer. On an RTX 5090 (PCIe 5.0 ×16, ~60 GB/s) moving ~140 GB of expert weights adds about 2 seconds; on RTX 4090‑class desktops (PCIe 4.0 ×16, ~25 GB/s) the same transfer costs ~5 seconds, and on typical laptop x8 links it exceeds 10 seconds.

Agentic tool calls repeatedly invalidate checkpoints, forcing the engine to re‑prefill large context blocks. Because checkpoints are sparse, each invalidation triggers a full re‑prefill of thousands of tokens, which on consumer GPUs occupies the accelerator for tens of seconds.

During decode, static expert placement leaves most routed experts on the CPU, while the GPU sits idle. Consumer CPUs deliver only ~50‑90 GB/s of memory bandwidth, far below the 1‑1.8 TB/s that a high‑end GPU can stream from its on‑package memory, creating a severe decode bottleneck.

The optimal split of work between GPU transfers and CPU execution depends on the hardware; a laptop with an RTX 4060 and LPDDR5 favors CPU execution, while a desktop RTX 5090 with DDR5 benefits from more GPU transfers. No universal rule exists, so each device must quantify the trade‑off on‑the‑fly.

On edge devices the VRAM budget is volatile because the GPU is shared with the desktop compositor, browsers, and games, causing the available memory to shrink or grow at any moment. Moreover, KV‑cache demand grows with each agent turn while the expert working set stays roughly constant, making a static split between cache and experts quickly suboptimal.

Starting a serving engine is costly: loading the ~140 GB expert pool from a 7 GB/s NVMe drive alone takes about 20 seconds before any warm‑up can begin, and this cost recurs each time the user opens the engine on an edge machine.

System Architecture

FreeToken reorganizes edge MoE serving with a two‑level memory hierarchy and semantic‑aware caching.

Edge‑serving MoE models run out of GPU memory because the full expert pool cannot fit. FreeToken solves this by introducing a two‑level memory hierarchy that keeps the authoritative expert pool on CPU and caches active experts on GPU.

The CPU holds the complete expert pool as the source of truth, while the GPU hosts a shared elastic cache that stores only the experts needed for the current forward pass.

How does this hierarchy differ from the naïve “offload whole layers to CPU” approach?

In the naïve approach an entire layer’s weights are swapped between CPU and GPU, incurring a full‑layer transfer each time the routing changes. FreeToken’s hierarchy keeps the full pool on CPU permanently and only moves the small subset of experts actually needed for the current token batch, reducing transfer volume dramatically.

During the initial “prefill” stage, FreeToken hides expert transfers behind ongoing computation and preserves recurrent state across context edits.

Buffer A holds layer 1 weights (4 MiB); Buffer B is empty.

GPU starts computing layer 1 using Buffer A while a PCIe stream begins loading layer 2 weights into Buffer B.

After layer 1 finishes, Buffer A is freed and the PCIe stream continues until Buffer A now contains layer 2.

GPU immediately begins computing layer 2 from Buffer A while layer 3 starts loading into Buffer B.

This ping‑pong continues, so the GPU never stalls waiting for weights.

By overlapping transfer with computation, the effective prefill latency is limited by the slower of compute or PCIe bandwidth, not by a serial transfer cost.

Why not simply load each expert on demand instead of double buffering the whole layer?

On‑demand loading would require a PCIe transfer for every expert, incurring many small transfers that underutilize bandwidth and stall the GPU. Double buffering moves a full layer once, allowing the transfer to be amortized over many experts and overlapped with compute.

During generation, cache misses are split between PCIe transfers and CPU execution based on measured bandwidths, ensuring the fastest path for each missing expert.

Miss count $m=2$. Compute split factor $q^{*}=2\cdot12/24=1$.

Since $q^{*}=1$, one miss ($e_{3}$) is transferred via PCIe, the other ($e_{4}$) is computed on the CPU.

PCIe transfer of $e_{3}$ completes in $4\text{ MiB}/12\text{ GB/s}=0.33$ ms.

CPU computes $e_{4}$ in $4\text{ MiB}/24\text{ GB/s}=0.17$ ms.

GPU merges the results from hits and the two miss paths to produce the final token.

The adaptive split ensures that the slower PCIe path is used only when its cost does not exceed the CPU compute cost, minimizing overall latency.

How does this bandwidth‑adaptive miss handling differ from a simple LRU cache that always transfers missing experts?

A pure LRU cache would always move a missing expert over PCIe, even when the CPU could compute it faster. FreeToken’s policy evaluates the relative costs and may choose CPU execution, reducing latency for small or cheap‑to‑compute experts.

**Figure 2** FreeToken overview. (1) Prefill: expert loading is double-buffered at full-layer granularity, streaming layer $l+1$ over PCIe while the GPU computes layer $l$; recurrent-state checkpoints are anchored at special-token boundaries, so a context edit resumes from the nearest surviving anchor and re-prefills only the new suffix. (2) Decode: most routed experts hit the shared LRU expert cache (here 8 of 12, following temporal locality). The $m=4$ misses are divided by $q^* = m \cdot B_P/B_H$ between cache fills over PCIe (one expert) and in-place CPU execution (three), using bandwidths profiled on the deployed machine; the GPU and CPU partial outputs merge exactly. The host-resident expert pool remains the source of truth throughout.

Semantic-Aware Expert Caching

We describe semantic‑aware expert caching and the optimal bandwidth‑adaptive split for decode.

During decode the router repeatedly selects a small working set of experts, exposing strong temporal locality that a static placement cannot exploit.

Instead of pre‑assigning experts to GPU memory, the cache follows the router’s real‑time demand, keeping the most recently used experts resident.

Compute $q^\star \approx m \frac{B_P}{B_H} = 3 \times \frac{4}{8} = 1.5$, round to $q = 2$.

Select $F$ (cache‑fill set) of size $2$; assume the two most recent missing experts are $e_1$ and $e_2$.

Remaining expert $e_3$ goes to CPU‑execution set $C$.

Transfer $e_1$ and $e_2$ over PCIe: time $T_{\text{fill}} = \frac{q S}{B_P} = \frac{2 \times 1}{4} = 0.5$ s.

CPU processes $e_3$ using residual bandwidth $B_{\text{res}} = B_H - B_P = 4$ GB/s: time $T_{\text{cpu}} = \frac{(m-q) S}{B_{\text{res}}} = \frac{1 \times 1}{4} = 0.25$ s.

The cache fills the two most recent experts, keeping the cache warm, while the remaining miss proceeds on the CPU without stalling the GPU pipeline.

Detect the set of missing experts $M$ for the current token.

Compute the optimal fill size $q^\star$ using the measured bandwidths $B_P$ and $B_H$.

Split $M$ into cache‑fill set $F$ (size $q$) and CPU‑execution set $C$.

Launch the CPU branch first to process $C$ using the residual bandwidth $B_{\text{res}}$.

In parallel, transfer $F$ into the GPU cache, update the LRU residency, and evaluate the combined GPU set $G = H \cup F$.

Merge the partial sums from CPU and GPU branches to produce the final MoE output for the token.

How does this caching differ from a standard LRU cache that simply evicts the least‑recently used expert?

Standard LRU bases recency on generic accesses, ignoring the router’s semantic decisions. Our cache updates recency only when the router selects an expert, and it also splits the remaining misses between GPU fill and CPU execution, so eviction is driven by model‑specific demand rather than generic memory pressure.

Elastic Memory Management

Implementation details of Elastic Memory Management and GPU‑resident caching.

Building on the semantic‑aware caching layer, this section details how FreeToken turns the memory hierarchy into a fluid, GPU‑centric runtime.

Think of the GPU’s VRAM as a stretchy backpack that expands or contracts as needed, while the CPU‑resident expert pool stays locked in a fixed locker – the backpack’s size only changes performance, never the contents.

Initial allocation: floor(5 GB / 0.5 GB) = 10 expert slots are provisioned.

A new request expands the KV cache to 4 GB, reducing free memory to 4 GB.

Re‑division: floor(4 GB / 0.5 GB) = 8 expert slots remain; the two excess slots are evicted.

At a later point the KV cache shrinks back to 2 GB, freeing 6 GB for experts.

Re‑allocation: floor(6 GB / 0.5 GB) = 12 expert slots are reinstated without restarting the engine.

This dynamic split guarantees that the system always fits within the current VRAM budget while keeping the most useful experts resident, and it does so without any downtime.

How does Elastic Memory Management differ from ordinary GPU memory paging?

Standard paging moves data in and out of GPU memory arbitrarily, potentially breaking the execution path. Elastic Memory Management keeps the CPU‑resident expert pool immutable and only reshapes the GPU cache layout, so the logical routing of tokens never changes and correctness is preserved.

Imagine a self‑service vending machine that decides which snacks to restock without a human manager, using a fixed list of slots and a single pass to pick the least‑recently‑used items.

The kernel scans the timestamps once and selects the two oldest slots (12 and 15) as eviction victims.

These two slots are overwritten with the newly fetched experts, updating their timestamps to the current step.

The remaining slots (20 and 30) stay untouched, preserving their recent‑use information.

Even if more than q experts miss, the kernel still only incurs a single pass over the cache, guaranteeing O(K) cost independent of the miss count.

Why isn’t this LRU cache the same as a classic LRU that scans the whole cache on every miss?

Classic LRU would require a full scan for each eviction, leading to O(N) work per miss. Here the kernel extracts the K oldest slots in one pass, and the miss path simply consumes up to q of those pre‑selected victims, making the cost O(K) regardless of how many experts actually miss.

FreeToken flattens all expert weights into a single “library shelf” (the FTW format) so that loading is just pulling a pre‑packed box off the shelf, and then chooses the right tool (GPU kernel or CPU SIMD) to read from that shelf.

The loader reads the first 8 KB chunk into host bank 1, filling experts E₁₁ and E₁₂.

The next 8 KB chunk is read into host bank 2, filling experts E₂₁ and E₂₂.

After both banks are populated, the memory regions are pinned for DMA, enabling direct GPU transfers without extra copies.

Because the layout already matches the runtime bank structure, no reshaping or per‑expert indexing is required, cutting load time dramatically.

How does the FTW format differ from a naïve checkpoint loading that parses each layer separately?

Naïve loading reads each layer’s tensors, discovers their shapes, and then repacks them into the runtime layout, incurring many small I/O operations and extra memory copies. FTW stores the weights already flattened and aligned to the final bank layout, so a single sequential read fills the host banks directly.

**Table 1.** Test systems. $B_P$ is the measured host-to-device expert-transfer bandwidth over PCIe; $B_H$ is the measured effective bandwidth of the CPU-side MoE expert kernel. On the three rented servers the CPU-thread and DRAM columns give container quotas.

Performance Evaluation

FreeToken delivers up to 2.3× higher decode throughput and sub‑44 s worst‑turn latency across diverse workloads.

FreeToken tackles the edge‑serving bottleneck by caching active experts on the GPU while streaming the rest from CPU memory, using semantic‑aware policies to keep transfer latency low.

FreeToken outperforms the strongest baseline on Qwen3.6‑35B‑A3B by up to 2.3× in decode throughput.

Achieves 77–83 tok/s versus 33–36 tok/s for the best baseline across all workloads.

**Figure 3.** End-to-end serving on the RTX 5090 across four workloads (1. AIME, 2. OpenCode+SWE, 3. Claude Code+SWE, 4.OpenClaw+Email/Cal) and two models (Qwen3.6-35B-A3B BF16 and DeepSeek-V4-Flash MXFP4) Top: decode TPS; bottom: mean TTFT (log scale). × marks configurations an engine cannot serve (Ollama and MoE-Infinity lack DSV4 support; MoE-Infinity provides no usable server for multi-turn agents).

**Figure 4.** (a) Prefill TPS versus prompt length (RTX 5090, Qwen3.6-35B BF16), with and without FreeToken’s pipelined full-layer loading. (b) Decode-time expert miss rate versus cache size (as a percentage of the expert pool) under the three engines’ placement policies, replayed on identical routing traces; lines are means over W1–W4, bands the min–max range.

**Figure 5** Coding-agent decode TPS across consumer GPUs (SWE issues via the OpenCode harness), Qwen3.6-35B-A3B. RTX 4060 laptop using NVFP4, the other Qwen3.6 columns BF16. The RTX PRO 6000 column is a separate demonstration: Qwen-GLM-5.2 (753B-A40B, NVFP4) on the math workload; Ollama is not run there. $\times$ marks configurations an engine cannot serve.

Related Work and Conclusion

We situate FreeToken among prior MoE serving systems and summarize its broader impact.

FreeToken builds on the premise that MoE expert pools exceed GPU memory, so it caches active experts on the GPU while streaming the rest from host memory using semantic‑aware policies.

In the classic approach the entire expert pool lives in host memory; when a token routes to an expert, that expert is transferred over PCIe to the GPU, incurring a latency spike for each miss.

Introduced a GPU‑resident expert cache for on‑device MoE inference, moving only a subset of experts onto the GPU.

Combined an LRU expert cache with speculative prefetching to anticipate future routing decisions.

Guides caching and prefetching using request‑level activation patterns extracted from runtime traces.

Sharpens the predictor that anticipates future routing, improving cache efficiency.

Improves expert‑prediction accuracy through a learned flow model.

Refines routing predictors to further reduce miss rates.

Empirical studies showing that expert caches can be effective across diverse MoE families.

Fetches reduced‑precision replicas of missed experts to lower transfer volume.

Substitutes low‑scoring experts with cheaper alternatives to cut bandwidth.

Skips low‑scoring experts entirely, trading accuracy for bandwidth.

Restructures and fine‑tunes the router to reduce the number of active experts.

Streams dense model weights at layer granularity for throughput‑oriented batch inference.

Layer‑wise weight streaming for dense models to improve throughput.

Splits neurons by activation statistics, requiring ReLU‑family sparsity and learned predictors.

Statically assigns whole layers to devices at load time for CPU‑GPU hybrid execution.

Similar static layer assignment strategy as llama.cpp.

Executes missed experts on the CPU rather than moving their weights.

Fast in‑place CPU expert execution using AMX‑optimized kernels.

Rebalances CPU/GPU queues with per‑step schedule simulation.

Executes every miss on the CPU, eliminating GPU stalls.

GPU‑centric serving substrate that underlies FreeToken’s runtime.

Dynamic attention scheduling inside captured CUDA graphs.

Attention scheduling framework that enables dynamic behavior within CUDA Graph capture.

Hierarchical KV‑cache tiered across GPU, host, and remote storage for long multi‑turn sessions.

Splits VRAM between expert weights and KV cache based on marginal latency value.

Elastic memory pool that rebalances at runtime based on workload.

Pages experts to prioritize KV cache capacity, integrating expert paging with cache management.

In conclusion, FreeToken demonstrates that once sparse activation makes MoE models feasible, the remaining challenge is orchestrating GPU, CPU, host memory, and interconnect as a unified inference platform.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers