Speech Models

past, present and the future

tokens

predict

render

Tortoise

Encode, Model, Decode

Box one

Encode

Waveform in, codes in the middle, waveform out.

A VQ-VAE with 8,192 codes

Box two

Model

The same codes, predicted one at a time.

GPT-2 style, 30 layers

Box three

Decode

Codes in, waveform out.

Diffusion, then a vocoder

Open models today · PlayAI · 2025

PlayDiffusion: edit one word, keep the rest

Encode

XLS-R

Speech becomes discrete XLS-R tokens; the span to change is masked out.

Model

Llama-style

A pretrained text transformer, made non-causal, fills the masked tokens from both sides.

Decode

BigVGAN

A custom-trained BigVGAN turns tokens into audio, with the original speaker’s embedding.

Open models today · Kyutai · July 2025

Kyutai TTS

Encode

12.5 Hz

Mimi, the streaming codec from Moshi: semantic and acoustic codes together.

Model

1.6B

A 1B backbone over time plus a 600M depth transformer over each frame’s codes.

Decode

Streaming

Text runs about 1.28 s ahead, so speech starts before the full text arrives.

The parallel

The same three boxes as a text LLM

Compared with text LLMs

Where speech is harder than text

The knowledge is in text

Proofs, code and most of science are written, not spoken. Audio alone hits a ceiling on world knowledge.

Text takes turns; speech overlaps

Chat is one message after another. Talk has backchannels, interruptions and meaningful pauses.

Many more tokens

About 3 text tokens per second of speech, but 100 Mimi tokens: longer sequences, costlier context.

Anyway, we merge with the LLM

One model now reads, sees, hears and speaks

GPT-4o

OpenAI · May 2024

One network for text, vision and audio; voice replies in 320 ms on average

Gemini

Google · Dec 2023

Trained multimodal from the start: text, images, audio and video

Qwen3-Omni

Alibaba · Sep 2025

A Thinker reads everything; a Talker speaks. 30B total, 3B active

MiniCPM-o 4.5

OpenBMB · Apr 2026

9B: SigLIP and Whisper encoders into Qwen3-8B; full duplex

The recipe

Trained in four stages, like a language model

Merged, but real time has a budget

A small model talks. A big model thinks.

Human turn gap

~200 ms

The usual pause before someone answers, across languages

What fits in the call

3B active

Qwen3-Omni: 30B total, 234 ms to the first audio

What thinks in the background

GPT-5.5

GPT-Live hands reasoning and tools to it, off the audio path

Where the compute goes

RL went from a rounding error to pretraining scale

10×

RL compute from o1 to o3, about four months apart

≈ 4×

growth of all frontier training compute per year

Stability and preference, measured

RL already pays off in speech

The catch: reward hacking

To cut WER, Seed-TTS learned to talk slower and over-clearly, and lost naturalness.

Others doing it

Qwen3-TTS (DPO, then GSPO), IndexTTS 2.5 and GLM-TTS (GRPO). Graders: ASR error, speaker similarity, emotion classifiers.

Full duplex: RL on timing

RL can also teach when to speak

Moshi, aligned on real chats

Kyutai · Jun 2025 · offline DPO

283K preference pairs from live users. An LLM judge flags bad answers and bad timing: cutting the user off, or not answering.

QA 36.1 → 39.2 · Safety 70.9 → 77.8

DuplexPO: RL only for timing

NTU, NTU Taiwan, NVIDIA · Jul 2026 · GRPO

A reward per event: start on time, backchannel on cue, yield to a barge-in. What it says stays put.

Turn delay 7.3 s → 0.24 s · Yields to barge-in 65% → 100%

What TTS has

TTS RL has a few numbers and one task

Graders

Mostly metrics

Environments

Read this line

Emerging

Grading conversation

Duplex graders exist: good proxies, still coarse

Heuristics you can reward

Takeover rate during the user’s pauses

Backchannel frequency, per second

Backchannel timing vs humans (JSD)

Response latency when taking the turn

After an interruption: relevance (LLM judge) and latency

Moshi on these heuristics

From one reward to the whole trajectory

Code RL grades every step. Voice grades one moment.

Rubric judge for SWE agents

ByteDance Seed · Mar 2026

36.6% → 42.6%

SWE-bench Verified: tests only vs. a rubric judge scoring every turn

SWE-TRACE

vivo · Apr 2026

pass/fail → rubric

Pass/fail treats every success alike; a per-issue rubric scores the whole run

SpeechGym, in voice

UIUC + Amazon · Aug 2026

16% → 99.6%

Training groups that carry signal: outcome only vs. a reward per tool call

The future

Three numbers cannot grade a conversation

What text has

Text RL has many graders and many worlds

Graders

What gives the reward

Environments

Where the model acts

Environments you can download

Code has thousands of environments. Voice has ??

7,780

MiMo-V2.6 RL environments: code, terminal, cyber, web dev and more

5.5K+

SmolDataEnvs code and data tasks, graded without an LLM judge

1 gym

SpeechGym (Aug 2026), 4 spoken domains. The rest are evals

Voice agents · what exists today

Mostly evals. We need environments to hill-climb.

RL environment

SpeechGym

UIUC + Amazon · Aug 2026

24% → 53%

τ-Voice pass@1 after GRPO. Reset, step, reward: two omni models talk through 4 spoken domains.

Its interface takes any text agent domain and any open omni model.

τ-Voice

Sierra · Mar 2026

≤ 38%

Voice with noise and accents; text gets 85%

Full-Duplex-Bench v2

Feb 2026

Live examiner

An AI examiner talks; an LLM judge scores

Jarvis Bench v0.5

Voice Arena · Sep 2026

vs humans

Agents vs people: job done, and sounding natural

Full-Duplex-Bench v3

Apr 2026

5 × 4

Tool calls with real disfluency, 4 domains

VoiceAgentBench

Krutrim AI · Oct 2025

5,757

Spoken tool calls in 7 languages

MP-Bench

Shih et al. · Sep 2026

≤ 22%

Multiparty: when not to speak?

Sources

Papers and reports behind the talk