SadTalker

SadTalker is the OpenTalker team's open-source audio-driven talking-face animation project, with a paper at CVPR 2023, licensed under Apache-2.0 (commercial-use OK), and written primarily in Python (PyTorch). Input a single portrait image + an audio clip; output a talking-face video with synced lip motion, natural head pose, and stylized expressions, supporting multi-language, multi-style, and reference-image-driven rendering. The ComfyUI-SadTalker node is the most widely used in the AI creation community.

Core capabilities center on "3D motion coefficients + stylized driving". Architecture: three-stage network — ExpNet (expression coefficient generation), PoseNet (head pose coefficient generation), Audio2Coeff (audio → coefficients mapping) — learns 3DMM (3D Morphable Model) motion coefficients; a 3D-aware face renderer then composites final video frames. Compared to Wav2Lip (mouth only) and First Order Motion (full-pose transfer), SadTalker balances "natural head movement + accurate lip-sync + stylized expressions". Input: single portrait (any angle, frontal works best) + WAV / MP3 audio of arbitrary length; supports reference image style transfer (anime / oil painting / old photo); supports still mode (mouth only) / full mode (head + mouth) / reference mode (reference-driven). Output: MP4 video at 25-30 FPS; optional 512x512 / 256x256 resolution. Ecosystem: ComfyUI-SadTalker node (community-wide); Windows / Linux / macOS all supported; Docker / Gradio Demo / CLI.

Use cases include: digital-human spokesperson video generation (corporate / educational / marketing), audiobook / podcast visualization, AI voiceover animation, and the final step of a virtual-avatar + TTS workflow. Hardware floor: runs on CPU demo (slow, 5-10 seconds per video second); GPU recommended (NVIDIA 6GB+ VRAM); model weights ~2GB. Onboarding: git clone → download pretrained weights to checkpoints/ → python inference.py --driven_audio <wav> --source_image <png> --result_dir <out>; ComfyUI users install the ComfyUI-SadTalker node and use it inside workflows.

Licensed under Apache-2.0, commercial use + patent grant + trademark disclaimer; the paper requires citing "SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio Driven Talking Face Animation (CVPR 2023)". VisLane's current lip-sync / digital-human tag associations are weak (2 + 1); SadTalker will push these to 3 + 2, reaching the Pills threshold. SadTalker is also the open-source counterpart to VisLane's already-listed HeyGen (commercial digital human). For further quality (more natural head motion, longer videos), combine with the pending LivePortrait (portrait animation SOTA) and MuseTalk (Tencent real-time lip-sync).