Get started with the basics of big data and Hadoop clusters and how to use Yarn, MapReduce, and HDFS in Hadoop clusters.

Become a Big Data Professional

This module offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of big Data. Throughout this module, we will have plenty of opportunities to get our hands-on experience with functioning Hadoop clusters.

We will start off by learning about the rise of big data as well as the different types of data like structured, unstructured, and semi-structured data. We will then dive into the fundamentals of big data such as YARN (yet another resource manager), MapReduce, and HDFS (Hadoop Distributed File System). By the end of this module, we will have the foundations in place to start working with big data.

Introduction to Big Data and Hadoop

## DataNodes

The Datanode stores the actual data. It stores the data blocks on its local filesystem and sends a block report to the Namenode.

The data for HDFS files will be stored in the directory specified as the value for the configuration setting `dfs.datanode.data.dir`. The default value of this setting is `${hadoop.tmp.dir}/dfs/data`. In our terminal setup the data directory is located at `usr/local/tmp/dfs/data/`. We'll upload a file to HDFS containing a string; the corresponding data is written to the local disk of the DataNode. Execute the commands in the terminal and examine the outcome. A screen-shot of the directory with the data appears below:

# DataNodes

The Datanode stores the actual data. It stores the data blocks on its local filesystem and sends a block report to the Namenode.

The data for HDFS files will be stored in the directory specified as the value for the configuration setting `dfs.datanode.data.dir`. The default value of this setting is `${hadoop.tmp.dir}/dfs/data`. In our terminal setup the data directory is located at `usr/local/tmp/dfs/data/`. We'll upload a file to HDFS containing a string; the corresponding data is written to the local disk of the DataNode. Execute the commands in the terminal and examine the outcome. A screen-shot of the directory with the data appears below:

# DataNodes

The Datanode stores the actual data. It stores the data blocks on its local filesystem and sends a block report to the Namenode.

The data for HDFS files will be stored in the directory specified as the value for the configuration setting `dfs.datanode.data.dir`. The default value of this setting is `＄{hadoop.tmp.dir}/dfs/data`. In our terminal setup the data directory is located at `usr/local/tmp/dfs/data/`. We'll upload a file to HDFS containing a string; the corresponding data is written to the local disk of the DataNode. Execute the commands in the terminal and examine the outcome. A screen-shot of the directory with the data appears below:

This lesson explains the role of Datanode in HDFS.

Hadoop

Yarn

Map Reduce

HDFS

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Conclusion

Datanode

DataNodes