MapReduce in Batch Processing
Explore the MapReduce algorithm used for batch processing massive data sets across multiple machines. Understand key steps like data splitting, mapping to key-value pairs, local aggregation, shuffling by key, sorting, and reducing to produce final results. Learn how frameworks like Apache Spark enable efficient distributed batch processing with simplified coding.
In this lesson, we will learn a popular algorithm that is used frequently to do batch processing on a huge volume of data. Google published this algorithm in 2004, and it was later adopted in many data processing systems, such as Apache Spark.
The MapReduce algorithm
We’ll first look at this algorithm with an example. First, let’s imagine the following scenario:
- You have all the text of a piece of classic English literature.
- You want to count the occurrence of each word in the whole text.
- The data is stored in some persistent storage.