Execution of a Spark Application
Explore the detailed execution process of a Spark application. Understand how SparkContext triggers jobs, the role of DAG and task schedulers, and task assignment priorities based on data locality. Gain insight into executor task handling and failure recovery to better grasp Spark's workflow.
We'll cover the following...
We'll cover the following...
Execution of a Spark Application
As discussed earlier, a Spark job is initiated when an action is performed. Internally, the SparkContext object’s runJob(...) method is invoked. The call is passed on the scheduler. The scheduler runs as part of the driver and has two parts:
- DAG Scheduler
- Task Scheduler
The DAG scheduler breaks a job into a directed acyclic graph (DAG) of ...