NVIDIA's open model transcribes 40 languages, labels speakers

NVIDIA has released Nemotron 3.5 Transcribe, an open-weight, promptable speech model that turns recorded audio into text and can also attribute speakers, translate, and answer questions about the recording. Most transcription tools produce a word-for-word record and stop there. A transcript can capture every spoken word and still leave people wondering who said what, how to translate it, or how to spell domain jargon. Multilingual calls, fast-paced meetings, and medical dictation each need different output. Transcribe puts these tasks in one checkpoint and lets developers choose the output with a natural-language instruction.

The model transcribes 40 languages, translates 31 of them to or from English, and attributes speakers in 13. It handles recordings up to an hour long and labels up to eight speakers. Developers can guide output with natural language, choose where their audio is processed, and adapt recognition to domain vocabulary and accents. The same recording can produce different results depending on the instruction. Examples from NVIDIA include “Transcribe this recording and label each speaker,” “Translate this audio into English,” and “What decisions were made in this meeting?” NVIDIA also published a demo in which the model captions a scene from the 1940 screwball comedy His Girl Friday, a film known for rapid, heavily overlapping dialogue, and labels the speakers, including the overlapping lines.

For terms the model may not know, NVIDIA recommends trying context before training. In a healthcare example, the prompt asks for the medical discussion to be transcribed in its original language, lists possible terms such as HbA1c and metformin, and instructs the model to use those terms only when the audio supports them. It also asks the model to preserve standard medical spelling and write numbers in conventional written form. When prompting is not enough, the companion Granary v2 dataset and NeMo Speech fine-tuning tools extend the same control to training. In a medical case study, fine-tuning cut word error rate (WER) on held-out speakers by 45.8% relative to the best prompt.

The model card includes recommended presets, including a reasoning setting for audio questions, a full prompt guide, and a task-by-language matrix. Language support differs by task, and audio question answering is trained in English. Word-level timestamps are also supported. NVIDIA advises spot-checking names, speaker changes, and answers against the recording.

Benchmark Results

NVIDIA reports results across several independent leaderboards. On the Hugging Face Open ASR Leaderboard, Transcribe ranks first among open-weight models for English, with a 5.33% average WER, and API models are excluded from that comparison. In the model card's Open ASR comparison, it posts a 4.62% mean WER across seven multilingual languages and a 10.98% mean WER on long-form audio, both the lowest among the evaluated open-weight models.

On the Artificial Analysis Speech-to-Text Leaderboard, which ranks models by its AA-WER index, Transcribe ties for first among open-weight models at 2.8% AA-WER. The model it ties has about 24B active parameters, while Transcribe has about 4B. AA-AgentTalk, a held-out voice-agent speech set that makes up 50% of the index, shows Transcribe at 1.8% WER versus 2.3% for the next-best open-weight model. The model pairs its speech encoder with a hybrid Mamba–Transformer mixture-of-experts language model. Only about 4B of its 30B parameters are active per token, which places it on Artificial Analysis's open-weight Pareto frontier of accuracy versus active parameters. On the all-model chart, it ranks ahead of several widely used proprietary speech APIs.

For speaker attribution, NVIDIA says Sortformer speaker cues reach the language model alongside the acoustic features. As a result, Transcribe can tag each turn during generation rather than relying only on a later step to attach speakers to words. On David AI's DAI-ASR-I18N combined transcription-and-speaker-attribution track, Transcribe ranks first among evaluated open-weight models and second overall. On the diarization track, which scores who spoke when across 21 languages, it has the lowest diarization error rate of the 14 systems tested, at 18.26%. NVIDIA's dedicated Nemotron 3 Diarization model is next at 18.38%. On VoiceArena's English full-session Diarization-Bench, Transcribe posts the lowest concatenated minimum-permutation WER (cpWER) of 14 systems, at 32.4%, and ranks first in both in-person and online settings. Its deletions are 8.32% of reference words, compared with a median of about 31%. NVIDIA also cites results from Ocular AI's Converse-STT evaluation, where Transcribe ranks as the top open model, and from Josh Talks' Voice of India leaderboard, which reports results across Indian languages.

Running the Model

Transcribe runs on a single NVIDIA GPU, from Jetson AGX Thor and DGX Spark to Blackwell data-center GPUs. On one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with NVFP4 precision, it processes 100 hours of audio in under three minutes. That figure was measured on 30-second clips at high concurrency. Developers can serve the checkpoint with vLLM or run it locally with NeMo-Speech.cpp. The model is built for recorded audio, not live streaming.

Agora, whose real-time voice and video platform powers live conversations in apps, used word-level timestamps in a classroom demo. Transcribe turned a recorded class into a transcript shown alongside the replay, so students can see what was said and when. Agora's blog describes how the team built it.

Granary v2: The Data Behind the Model

A promptable speech model needs training data with unambiguous formatting targets. Granary v2 supplies that. Built with NVIDIA NeMo Curator, its reusable pipeline prepared the multilingual speech used in Transcribe's training. The pipeline produces aligned transcript variants, task labels, and inspectable quality metadata. The public manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1. NVIDIA notes that these figures describe the dataset release, not Transcribe's full training mixture or its sampling weights.

For each recording, Granary v2 generates three transcript forms, which match the verbatim and clean styles developers can request from Transcribe with a prompt. For the sentence “I paid, uh, twenty-five euros,” the text-normalized verbatim target keeps the words as spoken. The inverse text normalization target with disfluencies writes “I paid, uh, €25.” The version without disfluencies writes “I paid €25.” Beyond transcription, the pipeline produces short audio captions, code-switching restoration of loanwords, speech-grounded question-and-answer pairs, contextual ASR biasing terms, and translations with automatic quality scores.

The pipeline has four reusable stages. The first, Audio Preparation (Metadata), segments raw or long-form audio, tracks speakers across the full recording, and tags acoustic events and language. Granary v2's sources were already segmented, so this release skipped that stage. The second, Transcription (ASR), routes each language to primary and recovery ASR models and falls back to the recovery model when the primary fails a hallucination check. Human reference transcripts help preserve trusted spellings while adding spoken forms and disfluencies. The third, Task Label Generation (Text), derives the transcript variants and checks character rate, hallucination patterns, and agreement between the model and human references. Only accepted records receive contextual-ASR terms and translations. The fourth stage, High Fidelity (HiFi), is cut off in the source article.

Key Features at a Glance

• Multilingual coverage: 40 languages transcribed, 31 translated to or from English, 13 with speaker attribution – one checkpoint replaces several separate tools.
• Long recordings and speakers: Up to one hour of audio and up to eight labeled speakers – suited to full meetings and panel recordings.
• Prompt-driven output: Natural-language instructions switch between transcripts, speaker labels, translations, and audio question answering – one model serves several workflows.
• Word-level timestamps: Supported, as shown in Agora's classroom demo where transcripts sit beside lesson replays.
• Efficient size: About 4B active parameters of 30B total – NVIDIA says this matches a model with roughly 24B active parameters on AA-WER.
• Single-GPU deployment: Runs on one NVIDIA GPU, from DGX Spark to Blackwell data-center systems – 100 hours of audio in under three minutes on an RTX PRO 6000 Blackwell Server Edition with NVFP4.
• Domain adaptation: Prompt hints first, then fine-tuning with Granary v2 and NeMo Speech tools – a 45.8% relative WER cut on held-out speakers in the medical case study.

Availability

Nemotron 3.5 Transcribe is available on Hugging Face and ModelScope. Granary v2 is available on Hugging Face and ModelScope and is part of the Nemotron speech collection. The NeMo Speech fine-tuning tools are on GitHub. The source article does not give pricing. Transcribe is serving-ready through vLLM and can run locally through NeMo-Speech.cpp.