Workflow Tools

Cron is useful for simple pipelines but runs into challenges when tasks have dependencies on other tasks that can fail. To help resolve this issue, where tasks have dependencies and only portions of a pipeline need to be rerun, we can leverage workflow tools. Apache Airflow is currently the most popular tool, but other open-source projects are available and provide similar functionality including Luigi and MLflow.