LTX 2.3 Lip Sync: Talking Video and Audio Sync

Most "AI lip sync" is a repair job. You generate video with one model, produce audio with another, then use a third tool to warp a mouth until it roughly matches. Every hop is a place for sync to slip, and it usually does.

LTX 2.3 approaches this differently, and the reason is architectural: it is described as an audio-video foundation model where a 14B video stream and a 5B audio stream exchange information through bidirectional cross-attention inside the same 48 transformer blocks. Audio is not bolted on afterward — it is generated jointly with the picture.

That design produces four distinct routes to talking video, plus a guidance parameter whose only job is audio-visual sync. This guide covers all of them. Sourced from the LTX-2 pipeline documentation, the multimodal guidance docs, and the official ComfyUI tutorial, checked July 2026.

Four routes, four different starting materials

Pick by what is already in your hands. That framing prevents the most common mistake — trying to force one route to do another's job.

IA2V: image + audio → talking video

ComfyUI's Image-Audio-to-Video template is the most direct path: supply a portrait and a voice track, get lip-synced video. It runs on ltx-2.3-22b-dev-fp8.safetensors and, like every native LTX-2.3 workflow, needs no custom nodes.

Practical notes that decide the result:

• Audio quality in equals lip-sync quality out. A clean, dry vocal track with minimal background noise gives the model an unambiguous signal. Music beds and room reverb do not help it find phonemes.
• The source image matters more than people expect. A clear, well-lit, front-facing face at reasonable scale gives the model something stable to animate.
• Portrait works natively. LTX 2.3 does 9:16 without cropping — useful, since most talking-head content is vertical.

LipDub: replacing dialogue in existing footage

LipDubPipeline is the more specialized tool, and its mechanism explains its constraints.

It uses IC-LoRA on a distilled checkpoint with a single lip-dub IC-LoRA applied in both stages. The reference clip supplies video and audio reference tokens whose VAE latents are appended to the target audio sequence as frozen reference tokens — that is how speaker identity survives the re-voicing.

What follows from that design:

• Frame count and frame rate come from the reference video, and the frame count is silently snapped to the nearest 8k+1. The CLI does not accept --num-frames or --frame-rate.
• Required: --reference-video. Optional: --reference-strength.
• --lora must be passed exactly once, with a lip-dub IC-LoRA. Not zero, not two.
• The LoRA is Lightricks/LTX-2.3-22b-IC-LoRA-LipDub.

Use it for dubbing, for rephrasing a line without a reshoot, and for matching lips to new audio while keeping the same speaker.

Want to hear the joint audio-video output first? Sync quality is the thing you cannot judge from a spec sheet. ltx23.app runs LTX 2.3 in the browser — generate a talking shot and listen before you build a local pipeline around it.

Audio-to-video: when the audio comes first

A2VidPipelineTwoStage generates video driven by an input audio file, and one design decision in it is worth calling out.

The documented flow: stage 1 generates video at half resolution with audio conditioning — video-only denoising with the audio frozen — then stage 2 upsamples 2× and refines the video while keeping the audio fixed, using a distilled LoRA.

And the detail that matters for production: the input audio is encoded via the audio VAE to seed the audio latent, but the original audio waveform is passed through and returned in the output to preserve fidelity.

Read that again if you work with music or recorded voice. Your audio is not re-synthesized and degraded — it comes back intact, with video generated around it. That is exactly what you want for music videos, podcast visuals, or any case where the audio is the asset.

Extra CLI arguments: --audio-path (required), --audio-start-time, --audio-max-duration — so you can drive a clip from a section of a longer track. Image conditioning and prompt enhancement are both supported.

ID-LoRA: the same person, every time

For a recurring presenter or character, prompt-level consistency will drift. ComfyUI's ID-LoRA template does personalization with voice cloning, using ltx-2.3-id-lora-talkvid-3k.safetensors on the dev FP8 checkpoint.

Identity and voice together is the useful part: a talking-head series needs the face and the voice to stay the same, and doing those in one mechanism avoids the drift you get from stitching two systems.

The parameter that exists only for sync

This is the section most LTX 2.3 tutorials do not have.

The MultiModalGuiderParams documentation defines a per-modality guidance term called Modality CFG:

modality_scale — steers the model away from unsynced video and audio results, improving audio-visual coherence. Set to 1.0 to disable.

A guidance term whose sole purpose is sync. Lightricks' own example configuration sets it to 3.0 for both the video and audio guiders when generating with audio, and the docs advise setting it to 1.0 (off) for video-only work.

The same example config reveals another useful asymmetry:

video_guider_params = MultiModalGuiderParams(
    cfg_scale=3.0, stg_scale=1.0, rescale_scale=0.7,
    modality_scale=3.0, stg_blocks=[29],
)
audio_guider_params = MultiModalGuiderParams(
    cfg_scale=7.0, stg_scale=1.0, rescale_scale=0.7,
    modality_scale=3.0, stg_blocks=[29],
)

Video CFG at 3.0, audio CFG at 7.0. The audio stream is steered more than twice as hard toward the prompt. If your generated speech sounds generic while the picture follows your prompt well, that gap is where to look.

So the debugging order for sync problems is: check modality_scale is above 1.0, then raise audio CFG, then improve the audio input — not "try more seeds."

Prompting for speech

Because audio and video share a prompt, dialogue is written into the shot description. The official quickstart example demonstrates the pattern inside a single paragraph: an incidental sound event (a sniff is heard), the voice quality and tone described, the dialogue lines in quotes, then a pause and further lines.

The rules that follow:

• Dialogue in quotation marks — these exact words are spoken
• Describe the voice: register, age, tone, pace, accent
• Name ambient sound — silence you did not ask for is a choice the model makes for you
• Sequence audio with action — "after a brief pause" places a sound in time

And keep the whole thing under 200 words, per Lightricks' prompting guidance.

Audio-only work is cheap

An underused workflow: T2AOneStagePipeline generates audio only — the video branch is absent, so only the audio modality is denoised and decoded through the audio VAE and vocoder, producing a wave file.

Which means you can iterate on voice — including testing an accent or voice-style LoRA — without generating a single frame. The trainer ships audio-focused configs (t2a_lora, v2a_lora, plus audio extension, suffix, and inpainting) for exactly this kind of work.

For anyone building a character voice, that is the cheapest possible iteration loop, and almost nobody uses it.

What the output actually is

Worth setting expectations: the vocoder is HiFi-GAN based, modified for stereo, upsampling a 16 kHz mel spectrogram to a 24 kHz waveform. Real synthesized stereo audio, not a sound-effect overlay.

FAQ

Does LTX 2.3 do lip sync? Yes, through four routes: the IA2V ComfyUI template, LipDubPipeline, A2VidPipelineTwoStage, and the ID-LoRA template for recurring identity.

Which is best for a talking avatar from a photo? IA2V in ComfyUI — image plus audio in, lip-synced video out.

Can I re-voice existing footage? That is what LipDubPipeline is for, using a reference video and a single lip-dub IC-LoRA on a distilled checkpoint.

Does my input audio get re-encoded? In audio-to-video, the original waveform is passed through and returned in the output to preserve fidelity.

How do I improve sync? Set modality_scale above 1.0 (Lightricks' example uses 3.0), consider higher audio CFG, and feed clean dry audio.

Can I test a voice without generating video? Yes — the text-to-audio pipeline generates audio only, with no video branch.

Why is my LipDub output a different length than the reference? Frame count is derived from the reference and silently snapped to the nearest 8k+1.

Bottom line

Lip sync in LTX 2.3 is not a post-process — it is what a jointly trained audio-video model does natively, with four routes depending on whether you start from an image, a clip, an audio track, or an identity.

The lever nobody talks about is modality_scale, a guidance term that exists purely to push output away from unsynced results. Set it, feed clean audio, write the dialogue into the prompt in quotes — and the pipeline that used to take three tools takes one.

To hear it before you build any of it, generate a talking shot at ltx23.app.

Sources

Verified July 2026 against primary documentation:

1. LTX-2 available pipelines documentation — LipDub, A2Vid, T2A pipeline behavior, arguments and constraints
2. LTX-2 multimodal guidance documentation — modality_scale, per-modality CFG example configuration
3. ltx-core package documentation — dual-stream architecture, Audio VAE, 24 kHz stereo vocoder
4. Lightricks/LTX-2 — official GitHub repository — prompting structure and example prompt with dialogue and sound events
5. ComfyUI official LTX-2.3 tutorial — IA2V and ID-LoRA templates, checkpoints and LoRA files
6. LTX-2.3-22b-IC-LoRA-LipDub — lip dubbing LoRA
7. ltx-trainer configs — a2v_lora, t2a_lora, v2a_lora, audio extension and inpainting
8. Lightricks/LTX-2.3 — Hugging Face model card — variants and frame/resolution constraints
9. LTX-2 installation & CLI flags — shared flags including --lora and --enhance-prompt
10. Lightricks/LTX-Desktop — audio-to-video in the desktop app
11. LTX-2 technical paper (arXiv 2601.03233) — joint audio-video architecture

Guidance values quoted are those in Lightricks' published example configuration; tune against your own material.