gallery-dl

gallery-dl is an open-source command-line image, gallery, and manga downloader maintained by mikf, first released in 2014, licensed under GPL-2.0 (copyleft), and written primarily in Python. It covers Pixiv, Danbooru, Twitter/X, Instagram, DeviantArt, Flickr, Reddit, Tumblr, NicoNico, Nijie, Konachan, Yandere, e-hentai, and 200+ more sites, automatically archiving user uploads, favorites, galleries, and tag-search results. Often described as "the yt-dlp of images", it pairs with yt-dlp for video to give a complete media-archiving stack.

Core capabilities center on programmable, batch gallery collection. Site modules: one extractor per supported site, internally extracting metadata (title, uploader, tags, date, resolution, file size) plus the main file URL, with consistent output as metadata.json plus the raw resource. Directory templates use Python format strings for fully customizable organization (e.g. {category}/{id}_{title}.{extension} or {user[name]}/{id}), auto-sorting by user / tag / date. Filter and dedup via --filter, --simulate, --download-archive enables incremental scraping, with XPath / JSON path extraction for page elements — ideal for dataset prep and versioned archives. Configuration via gallery-dl.conf supports per-site / per-user OAuth auth (Pixiv, Twitter), HTTP proxy, rate limiting, concurrency, archive mode, and resume. Metadata extension: --write-metadata exports to JSON / SQLite database, ready for downstream Python scripts to build dataset indices.

Use cases include: AI image training data preparation (batch compliant scraping from Pixiv, Danbooru, etc.), secondary-creation reference archive collection, researcher dataset prep (combined with dataset-tools), and hobbyist personal gallery backups. Hardware floor: a CLI tool, single-core CPU and a few MB of RAM, requires Python 3.6+; concurrency is limited by the target site. Onboarding: pip install gallery-dl → configure ~/.gallery-dl.conf (OAuth auth) → gallery-dl https://pixiv.net/users/12345; per-site config examples are in the conf directory.

Licensed under GPL-2.0 (copyleft); commercial closed-source products linking gallery-dl's code must publish derivative source. As a standalone CLI tool, there's no copyleft contamination. gallery-dl is widely recognized as the "yt-dlp of images" and shares the design philosophy of per-site extractors + metadata + archive — focused on images and galleries. For video+audio+image in one stack, pair with yt-dlp. Note: gallery-dl doesn't store downloaded content or proxy copyright judgment; users must comply with target platform ToS and applicable copyright law.