Hugging Face is adding support for running GGUF models efficiently in transformers, allowing users to load checkpoints sized for their laptop's memory through the familiar transformers APIs. Pick a GGUF from the Hub, load it with from_pretrained, and start generating on your own machine.

Running AI models on your laptop has become much easier, and llama.cpp has been a big part of that. Its inference engine powers local AI tools such as Ollama, LM Studio, and Jan. Alongside projects like MLX, it has helped make local inference a practical option for everyday use.

GGUF, developed by the llama.cpp team, is a widely used format for local inference. The team shares quantized checkpoints under ggml-org on the Hub. Publishers such as Unsloth, LM Studio Community, and bartowski also provide ready-to-use GGUF checkpoints in a range of quantizations. GGUF models have been downloaded millions of times.

To bring performance close to llama.cpp, Hugging Face is reusing its underlying ggml kernels through the kernels library and reducing overhead in generate. The initial focus is local inference on Apple Silicon, starting with the Qwen3.5 architecture.

GGUF packages model weights and metadata, including tokenizer information and an optional chat template, in one file. It supports different quantization levels, letting you trade some precision for a smaller memory footprint. Variants such as Q4_K_M mix tensor precisions, using mostly 4-bit weights while keeping sensitive tensors at higher precision.

Here's how quantization changes the file size of Unsloth's Qwen3.5-4B:

Hugging Face suggests starting with Q4_K_M, then trying Q5_K_M or Q6_K if you have more memory available. More aggressive quantization can help larger models fit, but the quality tradeoff depends on the model and the task.

Loading GGUF with transformers

To get started, you need:

• An Apple Silicon Mac
• A PyTorch version supported by the published ggml-quantization kernel builds, usually the two latest PyTorch releases
• The latest version of transformers (main for now, until the next release) and a compatible version of kernels

pip install -U "git+https://github.com/huggingface/transformers.git" kernels

To load a GGUF model, pass its Hub model_id and filename as gguf_file to from_pretrained.

No extra configuration is needed: when the weights stay packed on Metal, transformers automatically loads the compatible ggml/Metal layer kernels and uses ggml-org/ggml-attn as the attention implementation. If that kernel cannot be fetched, the model falls back to "sdpa" with a warning, and you can force "sdpa" by passing attn_implementation="sdpa" explicitly. See the GGUF documentation for more loading options.

import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "unsloth/Qwen3.5-4B-GGUF" filename = "Qwen3.5-4B-Q4_K_M.gguf" tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename) model = AutoModelForCausalLM.from_pretrained( model_id, gguf_file=filename )

Everything after loading uses the standard transformers API:

messages = [{"role": "user", "content": "Explain why the sky is blue in a few sentences."}] inputs = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", ).to(model.device) with torch.inference_mode(): outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Without a compatible quantization kernel, the loader falls back to dequantizing the model and uses more memory.

You can also use the same checkpoint with transformers serve, which exposes an OpenAI-compatible API:

pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

The model argument uses the format repository:filename—before the colon is the Hub repository (unsloth/Qwen3.5-4B-GGUF), and after it is the file to load (Qwen3.5-4B-Q4_K_M.gguf). This selects a specific quantization from a repository that may contain several.

For models whose chat template supports thinking, add --reasoning off to skip it or --reasoning on to enable it. The default, --reasoning auto, follows the chat template's default. See the reasoning options for details.

You can connect a client such as Jan or Pi by adding a custom OpenAI-compatible provider with these settings:

transformers runs the model on your Mac, while the client provides the conversation interface. The same endpoint can be used by other clients that support this API.

Performance benchmarks

Hugging Face's reference for local inference performance is llama.cpp. The comparison focuses on three GGUF checkpoints: a small dense model, a larger dense model, and a mixture-of-experts model.

The llama.cpp column comes from the llama-bench tool (build 5f55650a7, release b10200, Metal backend from ggml 0.18.0), run as llama-bench -m <model> -p 0 -n 128 -r 3, which reports tg128: the token-generation rate over 128 decoded tokens, averaged across three repetitions, with prompt processing excluded. The transformers column is generate producing the same 128 tokens from a 12-token prompt, best of three warmed runs, and includes prefill. Measurements were taken on a MacBook Pro M2 Max with 32 GB unified memory, macOS 26.6, PyTorch 2.12.1, kernels 0.17.0, plugged in.

transformers is close to llama.cpp across all three checkpoints. Note that the transformers measurement includes prefill while llama-bench reports decode-only throughput, so the measurements are not directly comparable.

Benchmark scripts

For transformers:

import time import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id, filename = "unsloth/Qwen3.5-4B-GGUF", "Qwen3.5-4B-Q4_K_M.gguf" model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename) tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename) inputs = tokenizer("The capital of France is Paris. The capital of Germany is", return_tensors="pt") inputs = inputs.to(model.device) with torch.inference_mode(): model.generate(**inputs, max_new_tokens=8, min_new_tokens=8, do_sample=False) torch.mps.synchronize() for _ in range(3): time.sleep(90) start = time.perf_counter() model.generate(**inputs, max_new_tokens=128, min_new_tokens=128, do_sample=False) torch.mps.synchronize() print(f"{128 / (time.perf_counter() - start):.1f} tok/s")

For llama.cpp:

llama-bench -hf unsloth/Qwen3.5-4B-GGUF:Q4_K_M -p 0 -n 128 -r 3

Use cases

When GGML and llama.cpp joined Hugging Face, the roles were described as complementary: llama.cpp provides a foundation for local inference, while transformers provides a foundation for model definition. GGUF support brings those two closer together.

llama.cpp remains the recommended engine when your priority is efficient local inference. Its dedicated runtime, memory management, and broad hardware support are built around that goal. GGUF support in transformers gives developers a convenient way to work with the same GGUF checkpoints inside transformers for several purposes:

• Experiment with GGUF in Python and PyTorch. Inspect intermediate activations with hooks, modify a model's forward pass, or prototype custom layers using familiar PyTorch tools.
• Evaluate GGUF models. Use existing transformers evaluation workflows to measure the quality of quantized checkpoints.
• Validate GGUF conversions. Loading the original checkpoint and its GGUF conversion in transformers makes it easier to check that the weights were converted correctly, accounting for quantization error.
• Try new decoding ideas. Use custom logits processors and stopping criteria with generate, or write your own generation loop in Python.
• Fine-tune from a GGUF checkpoint. Dequantize the weights and continue with a standard transformers training workflow.

For fine-tuning, use GgufConfig(dequantize=True):

import torch from transformers import AutoModelForCausalLM, GgufConfig model = AutoModelForCausalLM.from_pretrained( "unsloth/Qwen3.5-4B-GGUF", gguf_file="Qwen3.5-4B-Q4_K_M.gguf", quantization_config=GgufConfig(dequantize=True), dtype=torch.bfloat16, )

Broader opportunities

A bigger opportunity lies in bringing ggml's performance to models that llama.cpp does not support. transformers already provides the PyTorch implementations of these architectures. With ggml kernels and quantization schemes available in PyTorch, Hugging Face can work toward accelerating their supported operations without first implementing the entire model in llama.cpp. This is especially useful for new architectures, research models, and custom variants that may never receive a dedicated llama.cpp implementation.

This opportunity extends beyond the GGUF format itself. A kernel operates on tensors; it does not require the whole model to come from a GGUF file. The same building blocks can be integrated into other transformers models and loading workflows. This also opens a path to other modalities: computer vision models, audio models, and multimodal models could reuse compatible attention, normalization, and matrix multiplication kernels without first having a full implementation in llama.cpp. Each architecture still needs integration and validation; the initial GGUF examples cover text generation.

With the right kernels and an efficient generation loop, Python and PyTorch can deliver strong local inference performance. The kernels handle the heavy computation, while the generation loop keeps the GPU busy by avoiding unnecessary synchronization. Hugging Face's focus was to make eager execution fast without requiring torch.compile.