Most explanations of LTX 2.3 stop at "it is a 22B video model that also does audio." That sentence is true and tells you nothing about why it behaves the way it does — why audio lands in sync, why two-stage rendering is the default, why frame counts have to follow a strange arithmetic rule.
The answers are all in the architecture, and Lightricks documents it in unusual detail. The ltx-core package README describes the model as an asymmetric dual-stream diffusion transformer with 48 layers: a 14B-parameter video stream and a 5B-parameter audio stream, exchanging information through bidirectional cross-attention.
That single design choice explains most of what you experience using the model. This article walks through the engine component by component, and translates each piece into something you can act on. Sourced from the ltx-core documentation, the LTX-2.3 model card, and the multimodal guidance docs, checked July 2026.
Why "dual-stream" is the whole story
Most audio-capable video systems are pipelines: generate video, then generate audio from that video (T2V → V2A). Sync is something you repair afterward.
LTX-2 does not do that. The documentation describes it as jointly modeling the text-conditioned distribution of video and audio, capturing true joint dependencies rather than chaining two models. Concretely:
- Video stream: 14B parameters, handling spatiotemporal dynamics
- Audio stream: 5B parameters, handling a 1D temporal signal
- 48 transformer blocks shared between them, differing in width
- Bidirectional audio-visual cross-attention using 1D temporal RoPE — synchronization only, no spatial alignment
- Cross-modality AdaLN, where each stream's scaling and shift parameters are conditioned on the other stream's hidden states
The parameter split is worth sitting with. Video gets nearly three times the audio stream's capacity, which matches the information density difference — but audio is not an afterthought bolted on, it is a first-class stream inside the same 48 blocks.
What this means for you: when a lip-sync or sound-timing result is good, it is not luck, and when it is bad, more post-processing will not fix it — the fix lives in the prompt and the guidance settings that steer both streams.
Two VAEs, because video and audio are different shapes
Rather than forcing both modalities through one encoder, LTX-2 uses decoupled latent representations — separate modality-specific VAEs. The documentation lists the payoff explicitly: 3D RoPE for video versus 1D RoPE for audio, independent compression optimization, and native video-to-audio / audio-to-video editing workflows.
Video VAE: the encoder compresses pixels [B, 3, F, H, W] into latents [B, 128, F', H/32, W/32]. That is a 32× spatial compression into a 128-channel latent space, with temporal compression as well.
Look at that H/32 and the model card's rule suddenly stops being arbitrary: width and height must be divisible by 32 because that is the VAE's spatial compression factor. A dimension that does not divide by 32 has no clean latent representation. Same story for the 8k+1 frame rule — temporal compression works in blocks, and the +1 is the anchor frame.
Audio VAE: the encoder maps [B, mel_bins, T] to [B, 8, T/4, 16] — 4× temporal downsampling, 8 channels, 16 mel bins in latent space, roughly 1/25 s per token.
Vocoder: HiFi-GAN based, modified for stereo, upsampling a 16 kHz mel spectrogram to a 24 kHz waveform with doubled generator capacity for stereo output.
So the audio you get out is genuinely generated audio at 24 kHz stereo, not a stock-sound overlay.
Want to hear the dual-stream output rather than read about it? The synchronized-audio behavior is the part specs cannot convey. ltx23.app runs LTX 2.3 in the browser, so you can generate a talking shot and judge the sync yourself in a couple of minutes.
Gemma 3 as the text encoder — and why it produces two embeddings
LTX 2.3 does not ship its own text encoder; it uses Gemma 3 12B, downloaded separately. The ltx-core docs describe it as a multilingual encoder with multi-layer feature extraction and thinking tokens, producing separate embeddings for video and audio conditioning.
Two practical consequences:
- Your prompt is read twice, for two purposes. The same paragraph produces a video-conditioning embedding and an audio-conditioning embedding. That is why describing voice quality and ambient sound inside the prompt actually works — it is not being ignored by a video-only encoder.
- Multilingual capability comes from Gemma. The encoder is a general multilingual LLM, not a CLIP-style text tower.
It also explains the setup friction: you cannot run LTX 2.3 without downloading a separate 12B model, and the --gemma-root path is mandatory in the CLI pipelines.
The generation loop, end to end
The documented flow:
- Text prompt → Gemma 3 encoder → text embeddings
- Latent initialization in spatial format
[B, C, F, H, W] - Patchification — spatial latents converted to sequence format
[B, seq_len, dim]for the transformer - Denoising loop over 48 blocks, video and audio streams cross-attending each step
- Unpatchification back to spatial format
- VAE decoding to pixels — with optional latent upsampling first in two-stage pipelines
- Audio latents → Audio VAE → mel spectrogram → vocoder → waveform
Step 3 is where render cost is decided. Once latents become a token sequence, attention cost scales with sequence length — which is set by resolution and frame count together. That is the architectural reason the recommended pipelines generate at low resolution first and upsample latents in stage 2 rather than rendering high from the start.
Guidance: three signals, not one
Because there are two streams, guidance is richer than a single CFG slider. The MultiModalGuiderParams documentation lists per-modality parameters:
| Parameter | What it does | Typical range |
|---|---|---|
cfg_scale | Classifier-free guidance — prompt adherence | 2.0–5.0 (1.0 disables) |
stg_scale | Spatio-Temporal Guidance — temporal coherence via block perturbation | 0.5–1.5 (0.0 disables) |
stg_blocks | Which transformer blocks to perturb | e.g. [29] |
rescale_scale | Rescales guided prediction toward the conditional variance; limits over-saturation | 0.5–0.7 |
modality_scale | Modality CFG — steers away from unsynced audio and video | >1.0 to enable |
skip_step | Skip guidance every N steps for speed | 0 disables skipping |
modality_scale is the one nobody talks about, and it is arguably the most LTX-specific knob in the entire stack: a guidance term whose only job is audio-visual sync. Lightricks' own example config sets it to 3.0 for both streams when generating video with audio, and the docs say to set it to 1.0 (disabled) for video-only work.
The example config is also instructive on stream asymmetry: video guider at cfg_scale=3.0, audio guider at cfg_scale=7.0. The audio stream is steered harder toward the prompt than the video stream is.
What the architecture tells you to do differently
- Write audio into the prompt. There is a 5B stream and a dedicated embedding waiting for it.
- Enable modality guidance for talking shots. Sync is a guidance term, not a post-process.
- Respect ÷32 and 8k+1. They come from VAE compression, not from a validation whim.
- Generate low, upsample in latent space. Attention cost is a sequence-length problem.
- Treat audio-only work as cheap. With a dedicated audio stream and VAE, testing a voice through the text-to-audio pipeline costs no video compute.
- Do not expect a pixel upscaler to fix content. Stage 2 operates on latents while the model is still denoising, which is a different kind of tool.
FAQ
How many parameters does LTX 2.3 have? It is a 22B model; ltx-core describes an asymmetric split of a 14B video stream and a 5B audio stream across 48 shared transformer blocks.
Is the audio real generated audio? Yes — audio latents decode through an Audio VAE to a mel spectrogram, then a HiFi-GAN-based vocoder produces a 24 kHz stereo waveform.
Why must resolution be divisible by 32?
The Video VAE compresses spatially by 32× (H/32, W/32), so dimensions must divide cleanly.
What text encoder does it use? Gemma 3 12B, downloaded separately, producing separate video and audio conditioning embeddings.
What is modality_scale?
A guidance term that steers the model away from unsynced audio and video. Set above 1.0 for audio-video generation, 1.0 to disable for video-only.
Is this the same architecture as LTX-2? LTX 2.3 is a release in the LTX-2 family and shares the ltx-core architecture; the checkpoints and upscalers are 2.3-specific.
Bottom line
The LTX 2.3 video engine is best understood as two streams that talk to each other every layer, feeding two VAEs and a vocoder, conditioned by a general-purpose multilingual LLM. Every rule that seems arbitrary from the outside — the ÷32 resolutions, the 8k+1 frames, the two-stage default, the sync guidance term — falls out of that design.
Once you know the shape of the engine, the settings stop being folklore. And if you want to see the joint audio-video output before touching any of it, generate one at ltx23.app.
Sources
Verified July 2026 against primary documentation:
- ltx-core package documentation — dual-stream transformer, 48 blocks, 14B/5B split, VAE compression ratios, vocoder specs, Gemma encoder behavior
- Lightricks/LTX-2 — official GitHub repository — model description, packages, pipelines
- LTX-2 multimodal guidance documentation — CFG, STG, modality CFG parameters and example configs
- Lightricks/LTX-2.3 — Hugging Face model card — variants, ÷32 and 8k+1 rules, Gemma requirement
- LTX-2 available pipelines documentation — two-stage structure and pipeline list
- LTX-2 conditioning documentation — replacing vs guiding latents
- LTX-2 optimization documentation — quantization and compilation
- LTX-2 technical paper (arXiv 2601.03233) — architecture reference
- google/gemma-3-12b-it-qat-q4_0-unquantized — text encoder
- ComfyUI official LTX-2.3 tutorial — how the components map to workflow nodes
- ltx.io — official LTX site — product documentation hub
Architecture figures are those published in Lightricks' own package documentation; implementation details can change between releases.
