Prefect

Prefect orchestrates your data pipelines, ETL jobs, ML training runs, and scheduled tasks, handling failures intelligently. It's a scheduler that actually understands when things fail and knows how to retry, alert, and recover. Prefect's Python library is fully open source (Apache 2.0). You write normal Python functions, decorate them with @flow and @task, and Prefect handles scheduling, retries, logging, and dependency tracking. The open source server gives you a dashboard, API, and all core orchestration features.

Self-hosting the Prefect server is moderate effort. It's a Python app backed by Postgres. Docker Compose gets you running in 30 minutes. You'll need to maintain the server, database, and workers yourself. Prefect Cloud is where the paid tiers live: free tier gives you a managed server with limited features, Pro at $500/mo adds RBAC, audit logs, and service accounts. Enterprise adds SSO and custom retention.

Solo developers: self-host for free or use the Cloud free tier. Small teams: Cloud free tier works until you need RBAC. Growing teams: the $500/mo Pro tier is worth it when managing access across 10+ people costs more in time than money.

The catch: Prefect v2 was a major rewrite from v1, and the migration was rough. The ecosystem is stable now, but it burned some trust.