DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang

DSpark accelerates LLM inference by combining parallel draft generation with load-aware, confidence-scheduled verification.

How can we dynamically adjust the length of speculative decoding drafts based on real-time confidence and system load to maximize throughput?

Parallel drafters generate tokens quickly but suffer from rapid acceptance decay, while fixed-length verification wastes batch capacity on tokens likely to be rejected. DSpark introduces a semi-autoregressive architecture that uses a parallel backbone for speed and a lightweight sequential head to model inter-token dependencies. It then employs a confidence-scheduled verification mechanism that dynamically prunes draft blocks based on real-time system load and prefix survival probabilities. In production deployments, this approach accelerates per-user generation speeds by 60%–85% compared to previous production baselines, effectively shifting the Pareto frontier of LLM serving.

Paper Primer

Parallel drafters predict all tokens in a block independently, which causes "multi-modal collisions" where the model proposes incoherent sequences. DSpark fixes this by appending a lightweight sequential head—either a Markov or RNN-based transition bias—that allows each position to condition on previously sampled tokens within the block.

To prevent verification waste, DSpark uses a confidence head to estimate the survival probability of each draft token. A hardware-aware scheduler then uses these estimates to truncate draft blocks, ensuring the target model only verifies tokens that provide a positive expected return on system throughput.

DSpark significantly improves generation speed under production traffic.

Deployment within the DeepSeek-V4 serving system compared to the MTP-1 baseline. 60%–85% acceleration in per-user generation speeds at matched throughput levels.

DSpark outperforms both autoregressive and parallel drafters on accepted token length.

Offline benchmarks across Qwen3-4B, 8B, and 14B models. 16.3%–18.4% improvement over the parallel DFlash baseline.

Why does DSpark use a semi-autoregressive design instead of just making the parallel model deeper?

Deeper parallel models still suffer from independent prediction errors at later positions. The sequential head injects local transition information, which mitigates suffix decay and improves sequence coherence without significantly increasing drafting latency.

How does the scheduler handle the trade-off between verification accuracy and system throughput?

It uses a hardware-aware scheduler that treats verification as a global throughput maximization problem. By sorting all candidate tokens across all active requests by their survival probability, it greedily allocates target-model compute to the most promising tokens until the system's throughput capacity is reached.

The scheduler operates asynchronously using historical confidence estimates from two steps prior. This "dynamic top-K" selection hides scheduling latency and ensures compatibility with continuous CUDA graph replay, preventing the scheduler from stalling the GPU pipeline.

DSpark demonstrates that speculative decoding efficiency is not just about draft quality, but about dynamically matching verification compute to the current system load. For production engineers, this means moving away from static verification lengths toward load-aware, throughput-maximizing scheduling.

Introduction and Motivation

We expose the inefficiency of fixed‑length speculative decoding and motivate DSpark’s adaptive approach.

Large Language Models generate text autoregressively, so each token requires a full forward pass and inference latency scales with output length. Speculative decoding speeds this up by letting a lightweight draft model propose a block of tokens that a full‑size target model then verifies in parallel. However, the naïve fixed‑length approach leaves two major inefficiencies.

A draft model emits a candidate token block, and the target model checks the whole block at once, keeping the longest prefix that matches the target distribution.

First, parallel drafters predict each position independently, so they cannot capture intra‑block token dependencies; this causes acceptance to drop sharply for later tokens. Second, verifying every token in a long draft wastes batch capacity on tokens that are likely to be rejected, especially when many requests compete for the same hardware. Verifying every token is like checking every page of a book for typos when you already know most pages are fine—time is spent on pages that rarely need correction.

The fixed‑length speculative decoding pipeline is fundamentally inefficient because it over‑verifies low‑confidence tokens, limiting throughput in high‑concurrency serving.

Speculative Decoding Foundations

Key concepts behind speculative decoding and drafter designs.

Speculative decoding speeds up autoregressive generation by letting a lightweight draft model propose several tokens that the target model then verifies in bulk.

These drafters generate each token sequentially, conditioning on previously drafted tokens.

All $\gamma$ draft tokens are produced in a single forward pass, making drafting time almost independent of block size.

DFlash injects rich target‑model context into a parallel draft, allowing deep architectures and large blocks without extra drafting cost.

These background concepts set the stage for DSpark’s confidence‑aware scheduler, which dynamically truncates drafts to balance block length against verification success.

The DSpark Architecture

DSpark combines a semi‑autoregressive draft with confidence‑driven verification to cut wasted compute.

DSpark splits drafting into a fast parallel backbone and a lightweight sequential head, then uses a confidence head to prune low‑confidence suffixes before verification — this keeps draft latency low while only sending promising tokens to the expensive target model.

How does DSpark differ from classic speculative decoding?

Classic speculative decoding uses a fixed‑length draft and verifies the entire block, wasting compute on low‑confidence suffixes. DSpark dynamically truncates the draft based on per‑position confidence and system load, so only high‑probability prefixes are verified.

The parallel backbone is like a factory that produces all parts at once, while the sequential head is a quality inspector that checks each part in order, allowing the draft to stay fast yet coherent.

Parallel stage emits hidden states $\mathbf{h}_1,\mathbf{h}_2,\mathbf{h}_3$ and base logits $U_1,U_2,U_3$.

Markov head sees $x_0=D$ and $x_{1}=\text{of}$, computes bias $B(\text{of},\cdot)$ that boosts “course” and suppresses “problem” for position 2.

Sequential sampling draws $x_1=\text{of}$, then $x_2=\text{course}$ (thanks to the bias), finally $x_3$ is sampled from $U_3$ unchanged, yielding the coherent draft $[\,\text{of},\ \text{course},\ \text{...}\,]$.

The Markov bias eliminates incoherent cross‑position combos (e.g., “of problem”) while adding only a cheap matrix‑vector product per token.

Why not use a fully autoregressive drafter instead of the semi‑autoregressive design?

A fully autoregressive drafter would generate each token sequentially, incurring $T_{\text{draft}}$ that scales linearly with $\gamma$. The semi‑autoregressive design keeps $T_{\text{draft}}$ close to the parallel backbone’s cost while still providing enough dependency to raise the acceptance length $\tau$.

A lightweight confidence head predicts how likely each draft token will survive verification; a scheduler then picks the longest prefix whose expected acceptance maximizes overall throughput given current hardware load.

Compute prefix survival probabilities: $a_{1,1}=0.9$, $a_{1,2}=0.9\times0.6=0.54$, $a_{1,3}=0.54\times0.2=0.108$; similarly $a_{2,1}=0.8$, $a_{2,2}=0.4$, $a_{2,3}=0.04$.

Sort all $(r,j)$ pairs by $a_{r,j}$: (1,1)=0.9, (2,1)=0.8, (1,2)=0.54, (2,2)=0.4, (1,3)=0.108, (2,3)=0.04.

Assume the hardware throughput curve $\text{SPS}(B)$ peaks at batch size $B=5$ tokens. Greedy addition stops when adding the next token would increase $B$ beyond the peak without enough gain in $\tau$.

Result: schedule $\ell_1=2$ (accept first two tokens of request 1) and $\ell_2=1$ (accept only the first token of request 2), yielding $B=R+\ell_1+\ell_2=2+2+1=5$ and $\tau= a_{1,1}+a_{1,2}+a_{2,1}=0.9+0.54+0.8=2.24$.

The greedy selection automatically balances per‑request confidence against the global batch‑size‑throughput trade‑off, ensuring the system stays at the sweet spot of the SPS curve.

Greedy admission algorithm for hardware‑aware prefix scheduling.

**Figure 1 | The DSpark architecture and decoding cycle.** Given prompt tokens ABC, the target model executes one step to generate the next token D, which serves as the anchor for the drafting phase. Using D as the input, DSpark employs a heavy parallel backbone and a lightweight sequential head to generate draft tokens EFGH along with their corresponding confidence scores c1-c4. The Hardware-Aware Prefix Scheduler then evaluates these scores to retain the prefix EFG and drop the low-confidence token H. Finally, the target model verifies the scheduled prefix in parallel. As illustrated, E and F are accepted while G is rejected, prompting the model to generate a corrected token G* to complete the current round.

**Figure 2 | Position-wise conditional acceptance.** We report the empirical conditional acceptance rate for each draft position, averaged across benchmarks within each domain using the Qwen3-4B target model. Unlike standard prefix survival, this metric isolates the baseline predictive quality at position k by removing the penalty of previous rejections. Notice that the autoregressive drafter (Eagle3) remains stable or trends upward, while the parallel drafter (DFlash) suffers suffix decay.

Training the Confidence Head

Training shapes drafts to be accurate early, using a weighted multi‑loss objective.

Speculative decoding spends compute verifying low‑confidence tokens that were drafted early. To avoid that waste, the training regime must make drafts accurate where verification matters most—at the front of each block.

We weight the loss more heavily on early positions—like a teacher grading the first exam questions more strictly because they set the tone for the rest of the paper.

Cross‑entropy term: suppose the draft probabilities for the true tokens are $p_d^{1}=0.9$, $p_d^{2}=0.6$, $p_d^{3}=0.4$. Compute $L_{ce}=-(1\cdot\log0.9 + 0.72\cdot\log0.6 + 0.52\cdot\log0.4)\approx1.23$.

Total‑variation term: let target distributions be $p_t^{1}=(0.9,0.1)$, $p_t^{2}=(0.5,0.5)$, $p_t^{3}=(0.3,0.7)$ and draft distributions $p_d^{1}=(0.9,0.1)$, $p_d^{2}=(0.6,0.4)$, $p_d^{3}=(0.4,0.6)$. The $L_1$ distances are $0$, $0.2$, $0.2$ respectively, giving $L_{tv}=1\cdot0 + 0.72\cdot0.2 + 0.52\cdot0.2\approx0.25$.

Confidence term: assume soft acceptance labels $c^{*}_k$ equal the $L_1$ distances above (0, 0.2, 0.2) and the model predicts $c_k=(0,0.15,0.25)$. The BCE yields $L_{conf}\approx0.12$.

Overall loss: $L = 0.1\cdot1.23 + 0.9\cdot0.25 + 1.0\cdot0.12 \approx 0.31$.

The exponential weighting makes early positions dominate the loss, forcing the drafter to get the first tokens right—exactly where verification cost is highest.

Why not weight all positions uniformly instead of using the exponential decay?

Uniform weighting would treat early and late tokens equally, but verification only occurs for the early part of each draft. By decaying $w_k$, the loss concentrates learning where it matters most, improving the acceptance rate without wasting compute on tokens that are rarely verified.

Offline Performance Evaluation

DSpark delivers markedly higher accepted lengths and efficient verification across benchmarks.

DSpark achieves up to +30.9 % higher accepted length than the fully autoregressive drafter Eagle3 while matching or exceeding the parallel baseline DFlash.

Table 1 reports macro‑average accepted‑length gains of 30.9 % (Qwen3‑4B), 26.7 % (Qwen3‑8B), 30.0 % (Qwen3‑14B) over Eagle3, and 16.3 %–18.4 % over DFlash; the trend holds for Gemma4‑12B.

**Table 1.** Main speculative decoding results. We report accepted length ($\tau$) per decoding round (higher is better) for different target models and domains. Bold marks the best results.

**Figure 3.** Effect of drafter depth. With proposal length fixed, DSpark's performance improves as drafter layers are added. Notably, a shallow 2-layer DSpark outperforms a deeper 5-layer DFlash baseline, highlighting the parameter efficiency of sequential modeling.

**Figure 4 | Effect of proposal length and latency overhead.** DSpark consistently outperforms DFlash across various block sizes (left three panels). The rightmost panel demonstrates that the sequential head introduces minimal latency overhead during serving.

**Figure 5 | Confidence threshold sweep.** A threshold of 0 corresponds to standard fixed-length verification. As the threshold increases, the overall acceptance rate steadily rises because the confidence head effectively prunes tokens that would ultimately be rejected (hashed bars).

**Figure 6 | The Reliability Diagram on Alpaca Dataset.** While the raw confidence estimator achieves strong discrimination, its predictions are inherently overconfident. Applying post-hoc calibration helps to align the prefix survival probabilities with empirical acceptance rates. The shaded background histogram represents the frequency distribution of sample counts across different confidence bins.

Production Deployment and Scheduling

Deploying DSpark shows how each component shapes real‑world throughput and latency.

Speculative decoding wastes compute by verifying low‑confidence tokens in fixed‑length drafts; DSpark replaces the static draft length with a confidence‑aware scheduler that trims drafts dynamically, balancing draft size against verification success.

The scheduler watches the confidence scores of draft tokens and, given the current hardware throughput curve, picks just enough tokens to keep the GPU busy without over‑committing verification work.

How does this scheduler differ from a static confidence threshold?

A static threshold applies the same draft length regardless of load, so under high concurrency it wastes batch slots on low‑confidence tokens. The hardware‑aware scheduler instead tailors the draft length per request to the current throughput curve, discarding low‑confidence tokens early and freeing capacity for higher‑confidence work.

Training the draft model alongside the target model is made scalable by two system‑level tricks: (1) only hidden states, not full vocab logits, are exchanged between workers, reducing per‑token bandwidth to $O(d)$; (2) anchor‑bounded sequence packing groups independent draft anchors into dense batches while preserving causal masking via token‑level attention indices.

Post‑hoc STS calibration brings the confidence head’s expected calibration error down to roughly 1 %.

Reliability diagrams show the raw model’s ECE ranging from 3 % to 8 %, which drops to ≈1 % after STS.

**Algorithm: Hardware-Aware Prefix Scheduler** **Require:** Active requests $r \in \{1, \dots, R\}$; confidence sequence $c_{r,1}, \dots, c_{r,\gamma}$ per request; profiled step curve $\text{SPS}(B)$ **Ensure:** Selected per-request prefix lengths $\ell_1^*, \dots, \ell_R^*$ 1. **for** $r = 1$ to $R$ **do** 2. Compute prefix survival probabilities: $a_{r,j} \leftarrow \prod_{i \leqslant j} c_{r,i}$ for $j = 1, \dots, \gamma$ 3. **end for** 4. Construct candidate space $\mathcal{E} \leftarrow \{(r, j) \mid a_{r,j} > 0\}$ and sort descending by $a_{r,j}$ 5. Initialize states: $\ell_r \leftarrow 0$ for all $r$; Batch size $B \leftarrow R$; Expected accepts $\tau^* \leftarrow R$ 6. Initialize tracking: $\Theta_{\text{best}} \leftarrow R \cdot \text{SPS}(R)$; Selected lengths $\ell_r^* \leftarrow 0$ for all $r$ 7. **for each** $(r, j) \in \mathcal{E}$ in sorted order **do** 8. $\ell_r \leftarrow j$; $B \leftarrow B + 1$; $\tau^* \leftarrow \tau^* + a_{r,j}$ 9. Current throughput $\Theta \leftarrow \tau^* \cdot \text{SPS}(B)$ 10. **if** $\Theta > \Theta_{\text{best}}$ **then** 11. $\Theta_{\text{best}} \leftarrow \Theta$; Update selected lengths $\ell_r^* \leftarrow \ell_r$ 12. **else** 13. **break** 14. **end if** 15. **end for** 16. **return** $(\ell_1^*, \dots, \ell_R^*)$ achieving $\Theta_{\text{best}}$

**Figure 7 | Throughput vs. TPS.** Aggregate output token throughput against per-request generation speed (tok/s/user) under live traffic. In our production deployment, DSpark improves the observed throughput–interactivity frontier relative to the MTP-1 baseline under the measured traffic and engine configurations.

**Figure 8.** Load-adaptive throughput and verification budgets. Top row (a, b): Aggregate output throughput across varying levels of system concurrency. Bottom row (c, d): The average target verification budget allocated per request. As concurrent load increases, the dynamic scheduler automatically restricts the per-request verification length to prevent resource contention.

Related Work and Conclusion

Survey of speculative decoding, scheduling, and parallel generation, then brief definitions of DFlash and MTP‑1.

Related work clusters into three strands: speculative decoding algorithms, system‑aware scheduling, and parallel generation.

Algorithms that accelerate autoregressive generation by separating token proposal (drafting) from verification, often using rejection sampling to preserve the target model’s distribution.

Research that treats speculative decoding as a scheduling problem, adapting draft length and verification budget to real‑time system load and request priority.

Methods that generate multiple tokens simultaneously, either in a single pass (non‑autoregressive Transformers) or with limited sequential refinement.

DFlash is a blockwise speculative decoder that unrolls a fixed‑length draft chain in parallel, eliminating the sequential bottleneck of token‑by‑token drafting.

MTP‑1 is a confidence‑aware scheduler that truncates drafts dynamically, selecting the verification length that maximizes expected throughput.

The conclusion emphasizes that DSpark combines a semi‑autoregressive backbone with a lightweight sequential head and a hardware‑aware prefix scheduler, delivering substantial speedups in production while preserving generation quality.

Theoretical Caveats and Selection Bias

Counterexample demonstrates selection bias when early‑stopping is omitted.

This appendix presents a minimal counterexample that reveals selection bias when the early‑stopping condition of Algorithm 1 is removed. The bias violates the non‑anticipating property required for lossless speculative decoding.

We consider a single request ($R=1$) with maximum draft length ($\gamma=2$). The pre‑token confidence for the first position is $a_1=0.8$, and the profiled capacity curve is $SPS(1)=1.0$, $SPS(2)=0.5$, $SPS(3)=0.45$.

The expected throughputs for verifying zero and one draft tokens are $Θ_0=1\cdot SPS(1)=1.0$ and $Θ_1=(1+0.8)\cdot SPS(2)=0.9$, respectively.

Without early‑stopping, the scheduler proceeds to evaluate $Θ_2$ before any admission decisions are made. Because the Markov confidence head uses the previously sampled token, the next confidence $c_2$ depends on the realization of $x_1$.

Case 1: if $x_1$ yields a high continuation confidence $c_2=0.9$, then $a_2=0.8\times0.9=0.72$ and $Θ_2=(1+0.8+0.72)\times0.45=1.134$. The global maximum is $Θ_2$, so the scheduler returns $ℓ=2$ and admits the first token into the verification prefix.

Case 2: if $x_1$ yields a low confidence $c_2=0$, then $a_2=0$ and $Θ_2=(1+0.8+0)\times0.45=0.81$. The maximum remains $Θ_0=1.0$, so the scheduler returns $ℓ=0$ and the first token is not admitted.

Thus the admission decision for the first draft token dynamically depends on the token’s own sampled value, introducing selection bias: the scheduler preferentially admits tokens that lead to confident continuations.

To make the bias explicit, consider vocabulary $\{A,B\}$ with target distribution $p_t(A)=0.7$, $p_t(B)=0.3$ and draft acceptance probabilities $d(A)=d(B)=0.5$. The sum $\sum_{x}\min(p_t(x),d(x))=0.8$ matches the assumed $a_1$.

If $x_1=A$ the scheduler selects $ℓ=2$, admitting the draft token, which is then accepted with probability $\min(1, p_t(A)/d(A))=1$, so the output token is $A$. If $x_1=B$ the draft token is rejected, the target model draws a fresh token from $p_t$, yielding $A$ with probability $0.7$. The resulting output distribution is $Pr(Y=A)=0.5\cdot1+0.5\cdot0.7=0.85$, $Pr(Y=B)=0.15$, deviating from the target $(0.7,0.3)$ and proving the scheduler is not lossless.

Early‑stopping prevents this issue: because $Θ_1<Θ_0$, the scheduler halts immediately and returns $ℓ=0$ before evaluating any continuation‑dependent quantity such as $c_2$. The admission decision for the first position therefore depends only on pre‑token information, restoring the required non‑anticipating property.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers