Stable Diffusion

Stable Diffusion is a latent diffusion model released in 2022 by the CompVis lab at Ludwig Maximilian University of Munich (Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer). The accompanying paper High-Resolution Image Synthesis with Latent Diffusion Models was accepted as a CVPR 2022 Oral. Training was made possible by a generous compute donation from Stability AI and Runway, using a LAION image-text dataset subset. The code went open-source on GitHub on August 10, 2022 and triggered the entire open-source AI painting ecosystem: SDXL, SD3, Flux, ComfyUI, WebUI, and virtually every subsequent effort trace their lineage back to this work.

Core capabilities revolve around the latent diffusion architectural innovation. Unlike DALL-E 2 / Imagen which diffuse directly in pixel space, Stable Diffusion first uses a KL-regularized autoencoder to compress 512×512 images into a 64×64×4 latent, then performs the diffusion process there — substantially cutting VRAM and compute requirements. Concrete specs: an 860M-parameter UNet, a 123M-parameter CLIP ViT-L/14 text encoder, and a downsampling-factor-8 autoencoder; the model is pretrained on 256×256 then finetuned on 512×512. The repository ships four v1 checkpoints: sd-v1-1.ckpt (237k steps at 256² on laion2B-en plus 194k steps at 512² on laion-high-resolution), sd-v1-2.ckpt (515k steps at 512² on laion-aesthetics v2 5+), sd-v1-3.ckpt (additionally 10% text-conditioning dropout), and sd-v1-4.ckpt (same recipe, 225k steps). The reference sampling script bundles a Safety Checker (reducing the probability of explicit outputs) and an invisible-watermark (marking outputs as machine-generated). A HuggingFace diffusers integration is provided for daily inference, and the team expects future community development to concentrate on diffusers.

Use cases: cited as the methodological anchor in the vast majority of subsequent diffusion-model research papers; the source of model weights loaded by every major UI (A1111 WebUI, ComfyUI, Diffusers scripts); the base model repeatedly fine-tuned by LoRA / Textual Inversion / Hypernetwork / DreamBooth and friends. Note: the main branch has been inactive since June 2024 (pushed_at = 2024-06-18); subsequent SDXL / SD3 / SD3.5 development moved to Stability AI's own repositories and HuggingFace organizations outside CompVis. Hardware requirements: the scripts run in an ldm conda environment (conda env create -f environment.yaml) with PyTorch + transformers 4.19.2 + diffusers + invisible-watermark; at least 10GB VRAM for image generation.

Licensed under CreativeML OpenRAIL-M (an Open RAIL M license with use-based restrictions that ban specific misuse scenarios — illegal content, discrimination, generating false personal testimony — while preserving commercial use). The CompVis team explicitly notes in the README that the weights are research artifacts rather than products, and recommends adding additional safety mechanisms before deploying in production. Subsequent SDXL / SD3 / Stability AI commercial models use Stability AI's own commercial licenses, which differ from this repository's OpenRAIL-M.