ToonCrafter

ToonCrafter is a generative cartoon interpolation model jointly open-sourced by the Chinese University of Hong Kong (CUHK) and Tencent AI Lab. It was accepted to the SIGGRAPH Asia 2024 Journal Track and published in ACM Transactions on Graphics (TOG). The model addresses a long-standing pain point for animators and storyboard artists: filling in dozens of in-between frames between two keyframes by hand, which is both time-consuming and hard to keep consistent. ToonCrafter repurposes a pretrained image-to-video diffusion prior for the cartoon domain, then uses sparse-sketch guidance and reference-based coloring to automatically generate coherent in-between video between any two stylistically similar sketches or cartoon frames.

Core capabilities fall into three scenarios. First, cartoon keyframe interpolation: feed a start and end frame (512x320), and the model directly produces up to 16 frames of continuous animation. Second, sparse-sketch guidance: in addition to the bookend frames, you can supply one or more intermediate sparse sketches as motion cues, precisely controlling character direction and motion range. Third, reference-based coloring: feed a line drawing plus a reference image, and the model automatically colorizes the line art into a coherent animation. Model weights are open-sourced on Hugging Face at Doubiiu/ToonCrafter; the community has also produced an fp16-pruned fork that runs on 12GB of VRAM.

Use cases include filling in-between frames for animated shorts, previewing motion from a comic or storyboard, generating in-between frames for stylized short videos, and accelerating stop-motion workflows. Worth noting: the authors explicitly describe the project as an open-source research exploration rather than a commercial product, with no guaranteed success rate (README Disclaimer section, verbatim); there is no official commercial offering or web app, so be wary of clone sites that imitate the name.

Deployment relies on Python 3.8.5 with Anaconda; inference needs roughly 24GB of VRAM (about 24 seconds for 50 DDIM steps on an A100). The community has trimmed this down to the 10GB range. Two usage modes are provided: a command-line entry (sh scripts/run.sh) and a local Gradio demo (python gradio_app.py). Licensed under Apache-2.0; currently 6k+ stars and 500+ forks on GitHub. Note: the main repository has not seen frequent updates since 2025-03-19; subsequent work has shifted toward paper citations and community wrappers.