Get started with the basics of big data and Hadoop clusters and how to use Yarn, MapReduce, and HDFS in Hadoop clusters.

Become a Big Data Professional

This module offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of big Data. Throughout this module, we will have plenty of opportunities to get our hands-on experience with functioning Hadoop clusters.

We will start off by learning about the rise of big data as well as the different types of data like structured, unstructured, and semi-structured data. We will then dive into the fundamentals of big data such as YARN (yet another resource manager), MapReduce, and HDFS (Hadoop Distributed File System). By the end of this module, we will have the foundations in place to start working with big data.

Introduction to Big Data and Hadoop

## Rise of Big Data

In today's world, potential sources of data are all around us and easier than ever to capture digitally. Newspapers, a smart-phone's GPS location, restaurants we frequently visit, people we meet, our purchases, and when we wake-up and go to sleep, are all pieces of data that can be collected. Several technology advancements makes collecting, storing, and processing ever-increasing amounts of data possible. For example, sensors for a wide variety of tasks, like motion-detection, temperature, wind-speed and others, are cheap and have the capability to periodically emit data back to servers.  Smartphones have contributed immensely to the data volume being generated today. Because of faster speed and better network bandwidth, transmitting huge amounts of data is trivial. The drop in cost to procure disk storage and memory and the increased size of these storage mediums have contributed to new opportunities. They have the potential to solve problems that previously seemed too expensive or simply impossible.

# Rise of Big Data

In today's world, potential sources of data are all around us and easier than ever to capture digitally. Newspapers, a smart-phone's GPS location, restaurants we frequently visit, people we meet, our purchases, and when we wake-up and go to sleep, are all pieces of data that can be collected. Several technology advancements makes collecting, storing, and processing ever-increasing amounts of data possible. For example, sensors for a wide variety of tasks, like motion-detection, temperature, wind-speed and others, are cheap and have the capability to periodically emit data back to servers.  Smartphones have contributed immensely to the data volume being generated today. Because of faster speed and better network bandwidth, transmitting huge amounts of data is trivial. The drop in cost to procure disk storage and memory and the increased size of these storage mediums have contributed to new opportunities. They have the potential to solve problems that previously seemed too expensive or simply impossible.

This lesson talks about how data volumes have exploded in our world and why  that makes Big Data important.

Hadoop

Yarn

Map Reduce

HDFS

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Conclusion

Rise of Big Data

Rise of Big Data