MuseTalk

MuseTalk is the open-source real-time lip-sync model released by Tencent's TMElyra Lab in March 2024, with ★3.5k+ on GitHub, licensed under Apache-2.0 (commercial-use OK), and written primarily in Python (PyTorch). Positioning complements VisLane's already-listed SadTalker (audio-driven full-face animation) — MuseTalk focuses on the "mouth" single dimension without driving head pose, and uses latent space inpainting to implement reference face + audio → lip-synced video, achieving 30+ FPS inference on a single NVIDIA GPU, making it the de-facto standard for real-time livestream / real-time customer service / video meeting face-swap scenarios.

Core capabilities center on "real-time lip-sync + cross-identity generalization". Architecture: reference-based latent diffusion inpainting — input a reference face image + target audio → inpaint the mouth region of the reference face to generate audio-synced lip motion; does NOT modify head pose / expressions / eyes — single-point focus on the mouth. Inference speed: 30+ FPS on a single NVIDIA RTX 3090/4090 (512x512 resolution), real-time capable; supports batch inference; FP16 halves VRAM. Training data: Chinese + English multilingual audio, strong cross-identity generalization (no per-target fine-tuning needed). Input: single reference face (any angle) + arbitrary-length WAV/MP3 audio; Output: MP4 video at 25-30 FPS, 512x512 / 768x768 resolution options. Ecosystem: integrated with Whisper transcription + MuseTalk generation end-to-end pipeline; ComfyUI-MuseTalk node (community version); Windows / Linux / macOS all supported.

Use cases include: real-time livestream digital human (alongside LivePortrait for head pose), real-time video meeting face-swap (alongside OBS virtual camera), customer-service AI digital human lip-sync (alongside LLM dialogue + TTS), short-drama / short-video secondary dubbing lip replacement, and educational courseware personalization. Hardware floor: CPU runs demo (slow, 3-5s/frame), NVIDIA GPU recommended (8GB+ VRAM, RTX 3090+); for real-time streaming (30 FPS), RTX 4090 / A100 recommended. Onboarding: git clone → download pretrained weights to checkpoints/ → python inference.py --reference_image <jpg> --audio_path <wav> --output_video <mp4>; ComfyUI users install the ComfyUI-MuseTalk node.

Licensed under Apache-2.0, commercial use + patent grant + trademark disclaimer. Positioning versus VisLane's already-listed SadTalker (audio-driven full-face animation) / LivePortrait (portrait animation SOTA): SadTalker is "full-face driving" (audio → mouth + head pose + expression); MuseTalk is "mouth-only focus" (audio → mouth, no head movement); LivePortrait is "video-driven" (reference video → full-face animation, not audio). The three combine for "audio + pose collaborative" high-quality digital human: audio controls mouth (MuseTalk) + video controls head pose (LivePortrait) → perfect alignment; if you need head driving with one tool, choose SadTalker; if you need real-time streaming with mouth only, choose MuseTalk.