Scrapy

Scrapy is the most widely adopted industrial-grade crawler framework in the Python ecosystem. First committed in 2012, licensed under the permissive BSD-3-Clause, and written primarily in Python. It provides a complete Spider / Item / Pipeline / Middleware / Downloader architecture, suited for large-scale structured scraping and data processing pipelines. Widely used in data engineering, machine learning, and AI training data preparation.

Core capabilities center on an extensible, async scraping framework. Spider layer: users subclass scrapy.Spider and define scraping logic via start_requests + parse callbacks; prebuilt base classes include SitemapSpider, CrawlSpider, XMLFeedSpider; Selector API (CSS / XPath) integrates efficiently with lxml. Pipeline layer: Item Pipeline is a post-processing hook chain that can chain dedup, cleansing, validation, database writes, ML scoring, and other steps; mature unit testing + debugging. Concurrency model: based on Twisted async I/O + queue, a single machine can handle thousands of concurrent requests; AutoThrottle auto-tunes rate and concurrency. Middleware layer: customizable downloader middleware (proxy, UA, cookie, retry), spider middleware (request filtering, URL normalization); a mature extension ecosystem including fake-useragent, scrapy-rotating-proxies, scrapy-splash (JS rendering). Deployment and monitoring: Scrapyd (scrapy/scrapyd) is the official daemon with HTTP API for spider submission; commercial Scrapy Cloud (scrapinghub) offers cloud orchestration; open-source alternatives include SpiderKeeper / Gerapy. Ecosystem: scrapy-redis (distributed), scrapy-playwright (JS rendering), scrapy-impersonate (browser fingerprint spoofing) cover various edge cases.

Use cases include: large-scale data acquisition (price comparison, news aggregation), AI training dataset preparation (combined with gallery-dl / yt-dlp for media scraping), SEO and brand monitoring, researcher dataset prep, and enterprise ETL pipeline front-ends. Hardware floor: a Python async framework; single-core CPU handles 100+ concurrent requests, 8GB RAM can process hundreds of thousands of URLs per hour; high-concurrency scenarios scale horizontally with workers. Onboarding: pip install scrapy → scrapy startproject myspider → define Item / Spider / Pipeline → scrapy crawl myspider. Documentation is at docs.scrapy.org; community resources are rich.

Licensed under BSD-3-Clause (permissive), allows commercial closed-source use, modification, and redistribution; only requires preserving the copyright notice; no requirement to release derivative source — this is the license-friendliness delta versus gallery-dl (GPL-2.0) and yt-dlp (Unlicense). Scrapy, yt-dlp, and gallery-dl have complementary positions: Scrapy is for the "custom scraping logic + structured data post-processing" engineer scenario; yt-dlp / gallery-dl are for "per-site preset extractors + media archiving" user scenarios. For hybrid needs (some sites have existing extractors, others need custom crawlers), you can call yt_dlp / gallery_dl Python APIs inside a Scrapy project and inject them as Downloader Middleware. Note: Scrapy doesn't store scraped content or proxy copyright judgment; users must comply with target platform ToS, robots.txt, and applicable copyright law.