Video Models

One portrait plus a live audio stream: Kyutai open-sources AURA, a talking head at 20.7 fps on a single H100

November 22, 20260 views阅读中文原文
One portrait plus a live audio stream: Kyutai open-sources AURA, a talking head at 20.7 fps on a single H100

TL;DR

Kyutai — the Paris-based open research lab behind Moshi and Hibiki — released AURA (AUdio-dRiven streaming Avatar) on November 20: a family of speech-to-video models that turn one portrait photo plus a live audio stream into a talking-head video, one chunk at a time, without waiting for the whole clip to be spoken.

The two headline numbers:

  • AURA 1.3B: 576p at 20.7 fps end-to-end on a single H100 for 16 Hz playback — genuinely real time, and 23× faster than its own bidirectional teacher.
  • AURA 5B: 720p at 9.7 fps on one H100, 17× faster than its teacher (Snap's TalkVerse 5B) at roughly matching quality — but not real time.

Both sets of weights are non-commercial. That is the first thing to check before anyone plans a product on top of this.

Everything below comes from Kyutai's official blog post, the BMVC 2026 paper (arXiv:2611.26540), the official code repository and the Hugging Face model cards. Several independent outlets (AI Weekly, AI.info, TLDR and others) reported the same figures on November 20–21, which is what the cross-check against media coverage was for; no fact here goes beyond the primary sources.

The problem it actually solves

Until now, talking-head generation split into two unsatisfying camps.

The high-quality camp is slow. Models such as TalkVerse and Wan-S2V use bidirectional attention: they need several seconds of audio up front, then run roughly 50 denoising steps over a whole video chunk before emitting a single frame. The paper's own measurements on one H100: TalkVerse 5B runs at 0.58 fps end-to-end, Wan-S2V at 0.08 fps, Hallo3 at 0.22 fps. Ten seconds of video costs anywhere from tens of seconds to a couple of minutes. Fine for production work, useless for a conversation.

The real-time camp compromises on quality and, so far, has mostly covered text-to-video and image-to-video rather than audio. The few attempts that did add dense, time-aligned audio used 14B–18B parameter models that reached only about 2 fps on a single GPU — still outside real time.

AURA sits in between. It distils a 50-step bidirectional teacher into a 4-step causal student that generates chunk by chunk and needs only the current chunk's audio, with no multi-second look-ahead. Two consequences follow:

  1. Latency drops from "the whole clip" to "one chunk." A chunk in the paper model is 4 latent frames, about 1 second of video, which puts end-to-end delay at roughly 2 seconds. The official demo retrains with 2-latent-frame chunks (0.5 s) and brings that down to about 1.2 seconds, at the cost of running the demo at 480px.
  2. Memory stops growing with video length. A rolling KV cache keeps only the first chunk, used as an identity anchor, plus the most recent chunk, so footprint stays constant no matter how long the stream runs. The paper verifies this by generating 60 continuous seconds.

Which of the two to pick

  • AURA 1.3B: the teacher is a 1.3B bidirectional model trained from Wan2.1-T2V-1.3B with the TalkVerse pipeline, conditioned on Mimi, Kyutai's streaming audio codec — the same representation its voice assistant Unmute uses. Output is 576p at 16 fps. The merged checkpoint is about 6.3 GB in fp32. Weights are CC BY-NC 4.0.
  • AURA 5B: the teacher is Snap's TalkVerse-S2V-5B, itself built on Wan2.2-TI2V-5B, with wav2vec2-large-xlsr-53-english audio features. Output is 720p at 25 fps. The merged checkpoint is 5.7B parameters, 22.8 GB in fp32. The Snap Inc. Non-Commercial License applies, because the weights are distilled from TalkVerse.

Neither can be used commercially as shipped. The 1.3B model carries a non-commercial Creative Commons term; the 5B model inherits Snap's non-commercial research licence. Shipping a product means negotiating a licence with Kyutai and Snap, or retraining on your own data.

What the benchmarks say, including where it loses

On the HALLO3 test set (100 clips, 10 seconds each, English speech):

  • AURA 1.3B: FID 56.8, FVD 177.1, JEDi 0.403, CSIM 0.84, Sync-C 4.07, warping error 0.0036. Best FVD and best identity preservation in the comparison.
  • AURA 5B: FID 56.9, FVD 195.5, JEDi 0.313, CSIM 0.79, Sync-C 4.30, warping error 0.0032. JEDi, which catches discontinuities between chunks, is a notch better than the small model.
  • Hallo3 (14.5B): best FID at 42.8 and best JEDi at 0.180 — at 0.22 fps. The blog adds its own caveat: qualitatively, Hallo3 shows exaggerated mouth motion and recurring hand artifacts that these distribution metrics never flag.
  • Wan-S2V (14B): best lip-sync by a clear margin (Sync-C 5.42, Sync-D 9.0) — at 0.08 fps.

The loss is worth stating plainly: AURA does not win on lip-sync. 1.3B scores 4.07 and 5B scores 4.30 on Sync-C, both below Hallo3 at 5.06 and Wan-S2V at 5.42. For tight close-ups where every phoneme has to land, this is not yet the right tool.

There is also a human study worth reading: 27 raters, 400 pairwise comparisons, 100 HALLO3 clips. AURA 5B was preferred over Hallo3 on all four axes — visual quality, lip-sync, reference fidelity, naturalness — and against its 50-step teacher it ranked higher on reference fidelity and level on the other three, while being 17× faster. That says more about whether "causal, 4 steps" broke usability than any automatic metric does.

Does it hold up past a minute?

The real test for an avatar is not the first frame; it is whether the face drifts by second 40. Kyutai generated 60 seconds for each of the 100 references and scored every consecutive 10-second segment:

  • AURA 1.3B: FVD stays flat across the six segments (179–186, mean 181.3) and Sync-C holds at 4.7–4.8. Identity preservation slips from 0.826 to 0.783 on the mean, but the lab notes that a few drifting clips pull that average down; the median per-clip change over the minute is −0.02.
  • AURA 5B: FVD jumps from 182 to 195–206 after the first segment and then stabilises, while CSIM slides from 0.764 to 0.713. The larger model is the less stable one over long streams, even though its single frames look better.

That inversion is counter-intuitive but explicable: the 1.3B model has a weaker, smoother teacher and smaller chunk-to-chunk jumps, while the 5B chases detail and accumulates drift over long sequences.

How to use it: three steps

The repository is https://github.com/kyutai-labs/aura. It needs a CUDA GPU and uv — not Apple Silicon, not CPU.

  1. First step, clone and set up: git clone https://github.com/kyutai-labs/aura.git && cd aura && uv sync. On first run it pulls the final checkpoint, the Wan VAE and text encoder, and the matching audio encoder from Hugging Face.
  2. Second step, generate from the command line: uv run generate.py --image portrait.jpg --audio speech.wav --output out.mp4 runs 1.3B; add --model aura-5b for the larger model. The useful flags are --prompt for a short scene description, --num_chunks for length in 16-frame chunks, --seed, and --compile.
  3. Third step, wire it into a service: the Python API is streaming. Aura("aura-1.3b").generate(image, audio) yields tensors of shape [3, frames, H, W] in the range −1 to 1, one chunk at a time, ready to push into an encoder.

One gap you need to know about: the 20.7 and 9.7 fps figures come from an optimised inference pipeline. The reference code in the repository, with --compile, runs AURA-1.3B at about 16 fps (real time for 16 fps playback) and AURA-5B at about 8 fps on one H100. Matching the paper means optimising the pipeline to the same degree yourself.

Who it is for

  • Voice assistants that need a face. This is exactly what the official demo does: Unmute handles streaming speech recognition, an LLM writes the reply, streaming TTS speaks it, and AURA 1.3B turns that voice into a face in real time. AURA's 1.3B model is conditioned on Mimi, the same representation the TTS side uses, so the video follows the audio in the space it was trained on.
  • Long-form talking-head work such as streaming, support avatars or recorded lessons. The rolling cache keeps memory constant and 60 seconds of continuous generation held up; the lab only claims "60 s+", so anything much longer needs your own validation.
  • Research and non-commercial prototypes. Weights, code, and a dataset of 30K synthetic clips with precomputed teacher trajectories are all released — a rare complete package for work on distillation, streaming generation and lip-sync alignment.
  • Local or private deployments. One H100 is enough; no cluster required.

Not a good fit: what to avoid for now

  • Commercial products. 1.3B is CC BY-NC 4.0 and 5B is under Snap's non-commercial licence. "Open source" here does not mean Apache 2.0.
  • Work that demands strict lip-sync. Sync-C trails both Hallo3 and Wan-S2V; close-ups will show it.
  • Chinese, Japanese or any non-English speech. The paper states it plainly: training and evaluation cover English-speech-driven talking heads, and other languages, non-speech audio and non-portrait framings are untested. That is not "slightly worse" — it is "unverified".
  • 720p at real time. 9.7 fps against 25 fps playback is roughly 40 percent; for real time you have to drop back to 576p on the 1.3B model.
  • Sub-0.5-second conversational latency. With 0.5-second chunks the demo still sits at about 1.2 seconds. Going lower is an architecture change, not a tuning knob.

Pitfalls and caveats

  • Do not cut the step count. The ablation is blunt: 4 steps is the default. At 3 and 2 steps FVD barely moves, but face sharpness falls monotonically from 50.4 to 37.5 and 36.5; at 1 step FVD jumps to 229.2, sharpness drops to 27.0 and identity preservation falls to 0.65. Distribution metrics fail to see the blur that viewers see immediately.
  • Frame the input photo properly. Square or landscape portraits where the face fills a good part of the frame work best. The script resizes and centre-crops to the nearest training resolution, so a bad composition gets cropped badly.
  • The audio front ends differ. 1.3B uses Mimi, 5B uses wav2vec2. They are not interchangeable; aura/audio.py implements both separately.
  • Read the licences layer by layer. The code builds on TalkVerse and Wan2.2. The Wan VAE and T5 text encoder are Apache 2.0, Mimi and wav2vec2 carry their own terms, and only the distilled weights are non-commercial.
  • Do not use it to impersonate people. The repository has a dedicated responsible-use note: animate only portraits you have consent for, and disclose that the video is AI-generated.

Cost and time

  • Hardware: one H100 for real time. Training used 64 H100s, with LoRA only (rank 128) in both stages.
  • Memory: about 6.3 GB of merged weights for 1.3B and 22.8 GB for 5B, both fp32, before the VAE, text encoder and activations. Kyutai does not publish a peak-memory figure, so measure your own deployment.
  • Time: at paper settings, 10 seconds of video takes under a second on 1.3B. The reference implementation with --compile lands near 16 fps for 1.3B and 8 fps for 5B. First run also includes downloading weights from Hugging Face.
  • Money: the models are free to download; the cost is GPU time. Commercial licensing terms are not published and have to be negotiated.

Next step

  1. First step, try the hosted demo: https://aura.kyutai.org/ currently runs at 480px. Pick a face, click, and talk — decide whether a 1.2-second lag at 576p is acceptable before renting a GPU.
  2. Next step, reproduce it locally with your own portrait and a 10-second English clip. That validates CUDA, uv, weight downloads and input framing before any integration work.
  3. If this is headed for a product, start the licensing conversation early: ask separately about 1.3B (CC BY-NC) and 5B (Snap non-commercial), and confirm the terms of the Wan2.2, Mimi and wav2vec2 components.

Sources

  • Kyutai blog post: https://kyutai.org/blog/2026-11-20-aura/
  • Paper (BMVC 2026): https://arxiv.org/abs/2611.26540
  • Code repository: https://github.com/kyutai-labs/aura
  • Model weights: https://huggingface.co/kyutai/AURA-1.3B and https://huggingface.co/kyutai/AURA-5B
  • Dataset: https://huggingface.co/datasets/kyutai/AURA-Dataset
  • Live demo: https://aura.kyutai.org/

Comments

0 published
Log inorsign up to leave a comment.

Loading comments...