crawlee-python

Crawlee gives you a Python framework that handles the ugly parts: browser automation, request queuing, proxy rotation, and anti-bot countermeasures. It's what you build on top of instead of writing raw Selenium or Playwright scripts from scratch.

Apache 2.0. Built by Apify, who also sell a cloud scraping platform, but the library itself is fully independent. Supports both HTTP crawling (fast, lightweight) and browser-based crawling (Playwright under the hood for JavaScript-heavy sites). Automatic retries, request deduplication, and session management come built in.

Fully free to use. No gated features, no paid tier for the library itself. Apify's cloud platform is a separate product. You never need it. Solo developers get a production-grade scraping framework for $0. Teams of any size benefit from the built-in queue management and error handling that you'd otherwise build yourself.

The catch: this is the Python port of Crawlee, which started as a Node.js library. The Python version is younger and the ecosystem of plugins and examples is smaller. If you're in the Node world, the original Crawlee (also by Apify) is more mature. And Apify's cloud integration is deeply embedded; you'll see references to it everywhere in the docs even if you never use it.