Autoregressive decode generates one token per model forward pass, so single-request latency is bounded by how fast you can stream weights through the GPU — not by compute. Speculative decoding attacks that bottleneck by letting a small draft model propose several tokens that the large target model then verifies in a single parallel pass. We bring DFlash — a block-diffusion drafter — to AMD Instinct MI355X through vLLM on ROCm, benchmark it against Qwen3.5's built-in MTP drafter, and quantize the target model to mxfp4 to demonstrate that speculation and quantization stack.

The result: up to 5.02× single-request throughput on Qwen3.5-27B and consistent wins over MTP, with an mxfp4 target adding another ~10–20% on top.

Why DFlash?

Speculative Decoding: Amortize One Verify Pass Over Many Tokens

The win in speculative decoding is amortization: one expensive target verify pass commits multiple tokens. The more tokens the drafter can propose that survive verification (the acceptance length), the more the verify cost is amortized. The ideal drafter proposes deep blocks cheaply — exactly where autoregressive drafters struggle, because depth and cost are coupled.

Autoregressive Drafting Is Sequential and Drafting-Bound

Speculative decoding pairs a small draft model with the large target model. The drafter proposes several tokens, the target verifies them in one parallel forward pass, and every token that matches what the target would have produced is committed for free.

The catch is how the drafter generates its guesses. EAGLE-style drafters and the native multi-token-prediction (MTP) modules shipped with recent models are autoregressive: they emit draft tokens one at a time, so drafting cost grows with the number of tokens you want to speculate. That forces the drafter to stay shallow — capping draft depth and speedup.

DFlash: Block Diffusion + KV Injection

DFlash replaces the autoregressive drafter with a lightweight block-diffusion model. Two ideas make it work:

Block-diffusion drafting. Instead of predicting tokens one-by-one, the drafter starts from a masked block of length k and denoises all k positions together in a single forward pass. The number of draft tokens is decoupled from the number of draft passes — proposing a longer block is nearly free.

KV injection. Rather than re-encoding the context, DFlash writes the target model's hidden representations directly into the draft model's KV cache, so the drafter conditions on the target's rich features without recomputing them.

Figure 1. The DFlash architecture. A lightweight block-diffusion drafter denoises a whole masked block in a single parallel pass, while KV injection writes the target model's hidden states directly into the drafter's KV cache — so the drafter conditions on the target's rich features without re-encoding the context, and drafting cost stays roughly flat as the block grows.

How Much Does DFlash Alone Buy?

The clearest signal is acceptance length as a function of draft budget. As the budget grows (block size 4 → 8 → 16), MTP climbs but flattens, while DFlash keeps pulling ahead — because a bigger block is nearly free to draft.

Figure 2. Qwen3.5-27B · HumanEval · concurrency 1. MTP's throughput plateaus around 245 tok/s past budget 8 — each extra draft token costs another sequential pass — while DFlash keeps climbing to 396 tok/s at block=16.

Why Quantize the Target Model on Top?

DFlash accelerates the decode loop; weight quantization shrinks the target's memory footprint. They act on different bottlenecks, so the natural question is whether they compose — or whether a quantized target degrades the draft acceptance that speculative decoding depends on.

We re-ran both models with the target weights quantized to mxfp4 (the draft model unchanged). A quantized target is cheaper to verify (fp4 weights → ~¼ the HBM traffic in memory-bound batch-1 decode), and as long as it still agrees with the drafter about as often, DFlash keeps committing the same long blocks over a faster verify pass. The results show that is exactly what happens.

End-to-End Performance Results

Test Environment

Config mapping (per the DFlash model card): `t3` → MTP steps=3 / DFlash block=4, `t7` → steps=7 / block=8, `t15` → steps=15 / block=16.

Launching the Server

Common environment (shared across all runs):

MoE mxfp4 only: enable the native AITER fp4 MoE kernel (needs AITER >= 0.1.16.post2).
 
Without it, the MoE path falls back to a dequant-to-bf16 emulation that erases the gain.
 export VLLM_ROCM_USE_AITER=1

Common launch command (autoregressive baseline):

serve Qwen/Qwen3.5-27B  --host 127.0.0.1 --port 8000  --tensor-parallel-size 1  --max-num-batched-tokens 32768  --gpu-memory-utilization 0.9

DFlash — add a `–speculative-config` pointing at the block-diffusion draft model (`num_speculative_tokens` = 3 / 7 / 15 for block = 4 / 8 / 16):

serve Qwen/Qwen3.5-27B  --host 127.0.0.1 --port 8000  --tensor-parallel-size 1  --max-num-batched-tokens 32768  --gpu-memory-utilization 0.9  --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15, "draft_tensor_parallel_size": 1}'

MTP — the native Qwen3.5 multi-token-prediction path, no separate draft model:

--speculative-config '{"method": "mtp", "num_speculative_tokens": 15}'

mxfp4 target — point at the mxfp4 checkpoint and keep everything else identical (the draft model unchanged). For the dense 27B this resolves to a fused fp4 GEMM automatically; for the MoE 35B-A3B, `VLLM_ROCM_USE_AITER=1` selects the native fp4 MoE kernel:

serve Qwen/Qwen3.5-27B-MXFP4  ... same flags ...  --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15, "draft_tensor_parallel_size": 1}'

Benchmark Commands

We drive the server with the z-lab/dflash benchmark client, sweeping every `(dataset, concurrency)` pair:

-m dflash.benchmark  --backend vllm  --base-url http://127.0.0.1:8000  --model Qwen/Qwen3.5-27B  --dataset gsm8k  --num-prompts 128  --concurrency 1 # repeated for concurrency 32

Throughput = generated output tokens / wall-clock time. Speedup = config / autoregressive baseline at the same workload, concurrency, and precision. In throughput tables bold marks the fastest speculative config per row; in accept-length tables bold marks the higher value within each matched MTP/DFlash pair.

Results: Throughput and Speedup

For latency-bound single-request serving — where speculative decoding matters most — DFlash is the better config on every 27B workload (shown in Figure 3) and on workloads for 35B-A3B (shown in Figure 4). Speculative decoding has the least speedup on MT-Bench, consistent with results from the DFlash paper. DFlash is more beneficial for low-concurrency scenarios because at high concurrency the arithmetic intensity is high enough that the hardware is much more saturated. In this case, verification time of the target model increases significantly, damaging overall speedup.

Figure 3. Single-request (concurrency 1) speedup vs the autoregressive baseline. DFlash block=16 (dark orange) leads on every 27B workload and on the reasoning/coding workloads for the MoE 35B-A3B.

Figure 4. Batched (concurrency 32) speedup. DFlash block=16 still leads the high-acceptance workloads (GSM8K, HumanEval, MATH500), but at a full batch the verify step is already compute-bound, so on shorter-acceptance workloads (MBPP, MT-Bench) the smaller block=8 — or even MTP steps=3 — is the safer pick.

Results: Acceptance Length

Acceptance length is where block diffusion pulls away: going from budget 4 → 8 → 16, MTP climbs but flattens, while DFlash keeps rising. On 27B HumanEval, DFlash reaches 10.38 accepted tokens per target step versus MTP's 8.02.

Table 1. Qwen3.5-27B — mean accepted tokens / target step (concurrency 1).

Table 2. Qwen3.5-35B-A3B — mean accepted tokens / target step (concurrency 1).

Results: Quantizing the Target Model

This is the most interesting result: quantizing the target to mxfp4 stacks cleanly with DFlash. Acceptance is preserved and the cheaper verify pass lifts absolute throughput.

Note. On the dense 27B, mxfp4 resolves to a fused fp4 GEMM out of the box. On the MoE 35B-A3B the native fp4 MoE kernel must be enabled explicitly (`VLLM_ROCM_USE_AITER=1`, AITER ≥ 0.1.16.post2); otherwise vLLM falls back to a dequant-to-bf16 path that re-materializes the full expert stack every step and erases the gain. All 35B-A3B mxfp4 numbers below use the native fp4 path.

Acceptance length vs the bf16 target. Quantizing the target barely moves acceptance — every difference is within run-to-run noise. DFlash's draft quality does not depend on the target being full precision.

DFlash block=16, concurrency 1 — mean accepted tokens / target step

Per-position acceptance rate (marginal). Zooming into the shape of acceptance confirms it. The marginal accept rate at position i is `P(accept ≥ i+1 draft tokens)` — the survival probability of the i-th speculative token. Quantization does not degrade the curve shape.