Spark Application Lifecycle
Explore the Spark application lifecycle to understand each phase from submitting a job to its completion. Learn how SparkSession initializes the cluster, how driver and executor processes interact, and how tasks are executed across the cluster. This lesson helps you grasp the fundamental workflow of Spark applications and their management.
We'll cover the following...
Application life cycle
In this lesson, we'll track the various phases a Spark application goes through from the time it is submitted to when it is marked as completed.
-
As a user, the first step is to submit the Spark job to the cluster. Usually, this involves the user running the
spark-submitcommand in a terminal window. The command spawns a process that talks to the cluster manager. If YARN is being used as the cluster management software, then the client process connects to the ...