Muse Spark Open is the full-size model of the Muse series. Muse Glimmer, which Meta released in August, is a small dense model from the same series, ideal for running locally. Both models use the same tokenizer and vocabulary. The chat template follows the Muse Glimmer template, with additions for the new modalities. In Muse Spark Open, the input embedding and the output layer are separate matrices of about 1.66B parameters each.

Language Model Architecture

The language model has 62 layers and a hidden dimension of 8192. Each layer has 128 query heads and 16 key-value heads (grouped-query attention). Each head has 64 dimensions. The attention block has an output gate and an output normalization layer. The model also applies normalization to the queries and the keys (QK-norm). Each block combines its input and its output through a learned residual gate, not through a plain sum.

Muse Glimmer uses one dense feed-forward network in each layer. Muse Spark Open replaces this dense network with a Mixture-of-Experts (MoE) block. Each MoE block has 256 experts, and a router selects 8 experts for each token. Each expert is a feed-forward network that projects from 8192 to 4096 dimensions and back to 8192, then applies its own normalization to its output. The experts hold ~97% of the total model parameters.

Only 8 experts run per token, so the model computes each token with about 42B parameters. All 256 experts stay in memory, so deployment needs the memory of an 816B-parameter model, but it runs at the much faster speed of a 42B model. The BF16 checkpoint is about 1.5 TB in safetensors format. The NVFP4 checkpoint stores the parameters in 4-bit precision with negligible loss in accuracy, specifically for NVIDIA Blackwell GPUs.

In transformers, the BF16 checkpoint runs across 4 nodes of 8 H200 GPUs with tensor parallelism (TP=32), and the NVFP4 checkpoint runs on a single B200 node. If you want to deploy Muse Spark Open, check out the llama.cpp, vLLM, and SGLang recipes below.

Image and Video Encoder

A perception encoder converts images and video frames into tokens for the language model, just like Muse Glimmer. The encoder has 1.8B parameters, 50 layers, a hidden dimension of 1536, and 16 attention heads. It splits each image into 14×14 patches and produces features with 4096 dimensions. Position information comes from learned absolute embeddings and 2D RoPE. The feed-forward network in each layer uses LayerNorm and GELU. Every fourth layer and the last layer use full attention, and the other layers use sparse attention.

Muse Glimmer 30B uses an encoder with the same architecture. The design follows the Perception Encoder work from Meta. Video input goes through the same encoder as images.

Audio Encoder

Muse Spark Open accepts audio as input through its audio encoder. An audio tokenizer converts the audio into tokens, and the chat template marks the audio position with “ tokens. The encoder emits one audio token every 80 ms, which yields 12.5 tokens per second. So, for example, audio that is 10 seconds long produces 125 input audio tokens.

The encoder itself is a convolutional pyramid. Each stage consists of a patch embedder with a different patch size and 2 convolutional blocks of 1D kernels of size 7 with stride 1. These stages repeat for patch sizes of [16, 6, 5, 4]. At the end of the final stage, the temporal frames go to a final block of RMSNorm → linear projection → RMSNorm.

Context Window and KV Cache

The context window is 1M tokens at inference. The configuration sets max_position_embeddings to 1,048,576. This includes image, video, and audio tokens in the same window as text.

Grouped-query attention keeps the KV cache small for a model of this size. Each token adds 2 × 16 heads × 64 dimensions × 62 layers = 126,976 values to the cache. In BF16, this is about 250 KB per token and about 266 GB for a full 1M-token context.

Loading the Model and Processor

The Muse Spark Open implementation provides MuseSparkForConditionalGeneration for text generation from multimodal inputs, and MuseSparkProcessor for preparing conversations. The processor combines the tokenizer, image and video processors, audio feature extractor, and chat template.

Use a DistributedConfig to configure the parallelism settings. tp_size is the number of devices across which the dense layers are sharded, and ep_size is the number of devices across which the experts are distributed.

You can initialize the model and tokenizer as follows:

import os from transformers import AutoModelForMultimodalLM, AutoTokenizer, DistributedConfig distributed_config = DistributedConfig( tp_size=int(os.environ["WORLD_SIZE"]), ep_size=int(os.environ["WORLD_SIZE"]), ) tokenizer = AutoTokenizer.from_pretrained("meta-models/Muse-Spark-Open-816B-A42B") model = AutoModelForMultimodalLM.from_pretrained( "meta-models/Muse-Spark-Open-816B-A42B", distributed_config=distributed_config, ) messages = [ {"role": "user", "content": "Explain in one sentence why the sky is usually blue."}, ] inputs = tokenizer.apply_chat_template( messages, reasoning_effort="medium", add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device).to(model.dtype) input_len = inputs["input_ids"].shape[-1] outputs = model.generate(**inputs, max_new_tokens=2000)

Text Generation and Reasoning Effort

Prepare the conversation, then pass the messages to processor.apply_chat_template, enabling add_generation_prompt=True, tokenize=True, return_dict=True, and return_tensors="pt". Pass the processor output to model.generate, and decode the newly generated tokens, excluding the input prefix.

The processor has a lever for reasoning effort, mapping symbolic levels (low/medium/high/xhigh) to the integers the model accepts (16, 64, 256, 512). Turning off reasoning is not possible with Muse Spark Open, so keep the model at “low” reasoning if you want to use fewer tokens on an easier task. You can also set a numeric effort by including a line like Reasoning strength: 1 in the system prompt.

messages = [ {"role": "system", "content": "You should only answer with a number."}, {"role": "user", "content": "What is 17 * 23?"}, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", reasoning_effort="high", ).to(model.device).to(model.dtype) output = model.generate(**inputs, max_new_tokens=2000) generated_tokens = output[0][inputs["input_ids"].shape[1]:] print(processor.decode(generated_tokens, skip_special_tokens=False))

Image Input

The Muse Spark Open processor expands each image marker into the number of patch tokens required by the processed image, surrounded by image boundary tokens. This expansion keeps the text positions aligned with the visual features passed to the model. The processor also handles multiple images within a single prompt.

image_url = ( "https://huggingface.co/datasets/merve/vl-test-suite/" "resolve/main/pills.jpg" ) messages = [ { "role": "user", "content": [ { "type": "image", "image": image_url, }, { "type": "text", "text": "Do any of the components in this supplement interact?", }, ], }, ] inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, reasoning_effort="medium", return_dict=True, return_tensors="pt", ).to(model.device).to(model.dtype) input_len = inputs["input_ids"].shape[-1] outputs = model.generate(**inputs, max_new_tokens=2000) response = processor.decode(outputs[0][input_len:], skip_special_tokens=False) processor.parse_response(response)

Video Input

Video prompts include frame timing as well as visual content. The processor handles videos with audio in them, which is an upgrade from Muse Glimmer thanks to the audio encoder. The processor downsamples video frames and audio to align frames together. By default, videos are sampled at 2 FPS, capped at 96 frames. You can modify this by setting the fps of the video processor:

processor.video_processor.fps = 4.0

As a general rule, videos are frames in temporal sequence, and the more you downsample, the more you lose frames between specific events. Keep fps high if you want more accurate generations. There is a trade-off, though: higher fps requires more memory and increases prefill latency. The frames are interleaved with text input that states timestamps (e.g. “Time: 3.5s”). To pass in videos with audio, set load_audio_from_video=True in apply_chat_template.

Preserve the video's metadata when preparing inputs with return_metadata=True, so the processor can assign the correct framerate.

import torch messages = [ { "role": "user", "content": [ { "type": "text", "text": ( "Describe what happens in this video. " "Use timestamps and include relevant speech or sounds." ), }, { "type": "video", "url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/ramen.mp4" }, ], } ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", load_audio_from_video=True, return_metadata=True, ).to(model.device).to(model.dtype) with torch.inference_mode(): output_ids = model.generate( **inputs, max_new_tokens=256, do_sample=False, ) generated_ids = output_ids[:, inputs["input_ids"].shape[1]:] response = processor.batch_decode( generated_ids, skip_special_tokens=True, )[0] print(response)

Audio Preprocessing

The Muse Spark Open audio tower consumes a raw waveform. The processor expects mono audio at 24 kHz and pads the waveform to a multiple of 1,920 samples, corresponding to one audio embedding per 80 milliseconds. Audio supplied directly to the processor at another sample rate must first be resampled. You can do this with the load_audio util of transformers. load_audio uses torchcodec, librosa, or torchaudio backends, so make sure one of these is installed locally