Scrapy is the Python framework that most serious web scraping projects end up using. It's not a simple HTTP library. It's an entire crawling engine with request scheduling, middleware pipelines, built-in rate limiting, and export to JSON, CSV, or databases.
Fully free under BSD 3-Clause. nearly two decades of development, and the documentation is excellent. You define 'spiders' that describe how to navigate and extract data from sites, and Scrapy handles the concurrency, retries, and data pipeline.
The catch: Scrapy is async/Twisted-based, which has a learning curve if you're used to simple requests + BeautifulSoup scripts. JavaScript-rendered pages need Scrapy-Splash or Scrapy-Playwright as an add-on. And the biggest catch isn't technical: it's legal. Scraping at scale runs into rate limits, CAPTCHAs, IP bans, and terms of service issues. Scrapy gives you the tools but not the permission. For light scraping, requests + BeautifulSoup is simpler. For managed scraping with anti-bot handling, look at Crawlee.
Lightpanda is a headless browser built from scratch in Zig specifically for speed. We're talking 10-50x faster than Chromium-based headless browsers for page loading and JavaScript execution, at a fraction of the memory.
The target audience is anyone running browsers at scale: scraping pipelines, automated testing farms, AI agents that need to browse the web. When you're paying per minute of compute and per GB of RAM, a browser that uses 90% less of both changes your infrastructure costs dramatically. AGPL-3.0 license. They offer a cloud service alongside the open source browser.
The catch: it's not a full browser. It doesn't render pixels: no screenshots, no visual testing. JavaScript support is growing but not at Chrome-level compatibility. Sites with complex JS frameworks may not work correctly yet. And AGPL means if you modify it and serve it to users, you must open source your changes.
Crawl4AI extracts clean, structured data from websites for AI/LLM pipelines, handling JavaScript rendering, anti-bot measures, and output formatting automatically. Not a point-and-click scraper. A full framework for building crawlers that handle the hard stuff: request queuing, proxy rotation, browser fingerprinting, error recovery, and storage.
Supports three modes: plain HTTP requests (fast, for simple pages), Cheerio (HTML parsing without a browser), and full browser automation via Playwright or Puppeteer (for JavaScript-heavy sites). You pick the right tool for each job. Built by Apify, who run a web scraping platform. They open-sourced their crawler framework and it's legitimately good.
Completely free and Apache-2.0 licensed. No feature gates. You can deploy crawlers anywhere: your server, AWS, or Apify's cloud platform (which is paid but optional).
The catch: Crawlee is Node.js only. Python developers should look at Scrapy. Also, web scraping is inherently fragile; sites change, CAPTCHAs evolve, rate limits tighten. Crawlee gives you the tools to handle this but you still need to maintain your scrapers. Apify Cloud is the easy button for deployment ($49/mo+) but self-hosting works fine.
Crawlee gives you a Python framework that handles the ugly parts: browser automation, request queuing, proxy rotation, and anti-bot countermeasures. It's what you build on top of instead of writing raw Selenium or Playwright scripts from scratch.
Apache 2.0. Built by Apify, who also sell a cloud scraping platform, but the library itself is fully independent. Supports both HTTP crawling (fast, lightweight) and browser-based crawling (Playwright under the hood for JavaScript-heavy sites). Automatic retries, request deduplication, and session management come built in.
Fully free to use. No gated features, no paid tier for the library itself. Apify's cloud platform is a separate product. You never need it. Solo developers get a production-grade scraping framework for $0. Teams of any size benefit from the built-in queue management and error handling that you'd otherwise build yourself.
The catch: this is the Python port of Crawlee, which started as a Node.js library. The Python version is younger and the ecosystem of plugins and examples is smaller. If you're in the Node world, the original Crawlee (also by Apify) is more mature. And Apify's cloud integration is deeply embedded; you'll see references to it everywhere in the docs even if you never use it.
