Large language models have become the default interface for code generation, math problem solving, summarization, document understanding, and many other developer workflows. Under the hood, though, most LLMs still generate text one token at a time, with each token depending on all previous tokens. These autoregressive (AR) models consume their own outputs as they generate.

The autoregressive approach has been remarkably successful—it's stable to train, simple to serve, and responsible for much of the progress in modern language modeling. But it has a fundamental limitation: every new token requires a full model pass, and every weight must be loaded from memory before computation can start. For developers building latency-sensitive applications, running smaller batch sizes, or trying to maximize GPU utilization, token-by-token generation leaves performance on the table, as most GPU time is spent on memory operations rather than computation. Additionally, once a token is generated, it's final; autoregressive models cannot revise previous tokens, so mistakes can propagate during generation.

NVIDIA Nemotron-Labs Diffusion introduces a new approach: diffusion language models (DLMs) that generate multiple tokens in parallel, then iteratively refine them over multiple steps. These models better leverage modern GPU architecture, offer significant runtime performance benefits, can revise generated tokens for text editing and fill-in-the-middle tasks, and provide a built-in way to control inference budget—reducing refinement steps reduces compute requirements at runtime.

The Nemotron-Labs Diffusion family includes text models at 3B, 8B, and 14B scales under the commercially-friendly NVIDIA Nemotron Open Model License, plus an 8B vision-language model (VLM) under the NVIDIA Source Code License. NVIDIA is releasing both base models and instruction-tuned chat variants, along with training code via the NVIDIA Megatron Bridge framework.

Three Generation Modes in One Model

The core design principle: autoregressive and diffusion generation should be capabilities of the same model, not separate families. Nemotron-Labs Diffusion supports three generation modes:

• Autoregressive mode runs as a standard left-to-right LLM, maintaining compatibility with existing developer workflows.
• Diffusion mode generates block by block, gradually refining tokens over multiple steps.
• Self-speculation mode uses diffusion to draft multiple candidate tokens, then uses autoregressive decoding to verify them, combining diffusion-style drafting speed with AR verification reliability.

This flexibility matters for workloads with unpredictable batch sizes or single queries (batch size=1), where both speed and accuracy are critical. Selecting the inference mode is a deployment-time setting requiring almost no application-level changes, letting developers seamlessly switch between their current setup and Nemotron-Labs Diffusion.

Performance Gains

Nemotron-Labs Diffusion 8B achieves 1.2% higher average accuracy compared with Qwen3 8B. Measuring inference speed in tokens per forward pass (TPF), a hardware-agnostic efficiency metric: diffusion mode reaches 2.6× higher TPF than AR models, while self-speculation pushes that further to 6× for linear self-speculation and 6.4× for quadratic self-speculation, with comparable accuracy across evaluated tasks.

Technical Foundation

Diffusion language models have been promising for years but faced practical barriers: lower accuracy than strong AR models, difficult training, and limited KV-cache compatibility. Recent work changed that. Efficient-DLM demonstrated that pretrained AR models can be converted into diffusion language models through continued pretraining and block-wise attention mechanisms, preserving AR capabilities while enabling KV-cache-friendly parallel decoding.

Nemotron-Labs Diffusion builds on this insight: adding diffusion capabilities to an existing AR model. The model was trained with a joint AR and diffusion objective, retaining what it learned during initial AR training while diffusion added parallel drafting capability. It was pretrained on 1.3T tokens from the NVIDIA Nemotron Pretraining datasets and underwent supervised fine-tuning on 45B tokens from the NVIDIA Nemotron Post-training datasets.

Deployment Integration

Deployment support will soon be available in the main branch of SGLang. Currently, inference support is available through this GitHub pull request.

The integration lets you serve the same checkpoint three ways via a single configuration line:

• Plain autoregressive (`ar_mode=true`): The model behaves like any other causal LM, useful as a correctness reference or sanity check.
• Diffusion mode (FastDiffuser): The headliner for raw throughput. The model fills in 32-token blocks by iteratively denoising them, with a confidence threshold determining which tokens are “good enough” to commit each step.
• Self-speculation (LinearSpec): Drafts a block bidirectionally, then verifies it causally; any matching prefix gets committed. Output matches AR at temperature 0, with approximately 865 tokens/second on B200 on the speedbench dataset—roughly 4× the autoregressive baseline on the same hardware.

Getting Started

Developers can explore the model family on HuggingFace, read the technical report, and try the training recipe.