Perks of Apache Spark
Let's explore the benefits provided by Apache Spark.
Keeps application running
Spark provides graceful degradation in cases where memory is not enough so that the application does not fail but keeps running with decreased performance.
For instance, Spark can recalculate any partitions on demand when they don’t fit in memory or spill them to disk.
Increasing performance
Wide dependencies cause more data to be exchanged between nodes compared to narrow dependencies, so performance is increased significantly by reducing wide dependencies, or the amount of data that needs to be shuffled. One way to do this is by pre-aggregating data, also known as map-side reduction.
Note: As explained previously, the map-side reduction is a capability provided in the MapReduce framework through combiners.
Example
The following code performs a word count in two different ways:
- The first one will send multiple records of value 1 for each word across the network
- The second one will send one record for each word containing the number of occurrences.
Get hands-on with 1200+ tech skills courses.