airflow.tar.gz

LiveJobApp

Apache Airflow has become the dominant and ubiquitous Big Data workflow management system, leaving Oozie and other competitors miles behind in terms of features and functionality. 

In this course, you will learn the fundamentals of Apache Airflow starting with DAGs, DagRuns, Tasks and Task Instances - the building blocks of this popular technology. You’ll then move on to understanding the architecture, creating operators and sensors, and understanding XCom.

In the latter half, you’ll learn about branching, scheduling, and working with the UI. By the end of this course, you will have a new, highly coveted, and transferable skill to add to your resume.

An Introduction to Apache Airflow


As we write more complex workflows, a need arises to create branches within our workflows. Depending on the evaluation of some user-specified conditions, the workflow may execute a different set of tasks. Airflow provides operators to enable branching in workflows.

## Branching

Consider the simple workflow below. We have `task1` execute first, followed by `task2`, and, after that, the workflow branches and either executes `task3` or `task4`. We can use the `BranchPythonOperator` to implement our desired workflow.  Here is an illustration depicting the workflow:



As we write more complex workflows, a need arises to create branches within our workflows. Depending on the evaluation of some user-specified conditions, the workflow may execute a different set of tasks. Airflow provides operators to enable branching in workflows.

# Branching

Consider the simple workflow below. We have `task1` execute first, followed by `task2`, and, after that, the workflow branches and either executes `task3` or `task4`. We can use the `BranchPythonOperator` to implement our desired workflow.  Here is an illustration depicting the workflow:


This lesson explains how DAGs can incorporate branching logic within their workflows.

__default

Branching

Branching