Running Spark Applications
Explore how to run Spark applications by creating SparkSession and SparkContext programmatically. Understand different deployment modes like client and cluster on YARN, and learn to monitor jobs using the Spark History Server for effective Spark job management.
We'll cover the following...
Running Spark Applications
In previous lessons, when we fired-up the spark-shell, we interacted with an object of type SparkSession, represented by the variable spark. Starting with Spark 2.0, SparkSession is the single-unified entry point to manipulate data with Spark. There’s a one-to-one correspondence between a Spark application and a SparkSession. Each Spark application is associated with one SparkSession. SparkSession has another field:SparkContext which represents the connection to the Spark Cluster. The SparkContext can create RDDs, accumulators, broadcast variables and run code on the cluster.
The illustration below shows how Spark interacts with and runs jobs on a Hadoop cluster.
Next, we’ll rewrite the example executed on the spark-shell to count the car makes as a Java application. We’ll explicitly create the SparkSession and the SparkContext programmatically before writing the core ...