Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

data.tar.gz

HADOOP_HOME

JAVA_HOME

HDFS_NAMENODE_USER

HDFS_DATANODE_USER

HDFS_SECONDARYNAMENODE_USER

YARN_RESOURCEMANAGER_USER

YARN_NODEMANAGER_USER

HADOOP_CONF_DIR

ZK_HOME

PIG_HOME

AvroWriteExample

AvroReadExample

AvroGeneratedCodeReadExample

AvroGeneratedCodeWriteExample

AvroRPCExample

ParquetReadExampleJob

ParquetWriteExampleJob

ParquetAvroReadExampleJob

ParquetAvroWriteExampleJob

ParquetProjectionReadExampleJob

SequenceFileReadExampleJob

SequenceFileWriteExampleJob

SequenceFileSyncPointExampleJob

TestCarMapperJob

TestCarReducerJob

CarCounterMrProgramJob

MyLiveAppJob

DataNodeWebUI2

YarnWebUI

YarnWebUI-copy

YarnWebUI-copy-copy

JHS-UI

Spark-UI-copy

Spark-History-Server-UI-3

This course offers a one-of-a-kind rich and interactive experience to learn the fundamentals and basics of Big Data. Throughout this course, you will have plenty of opportunities to get your hands dirty with functioning Hadoop clusters.

You will start off by learning about the rise of Big Data as well as the different types of data like structured, unstructured, and semi-structured data. You will then dive into the fundamentals of Big Data such as YARN (yet another resource manager), MapReduce, HDFS (Hadoop Distributed File System), and Spark.

By the end of this course, you will have the foundations in place to start working with Big Data, which is a massively growing field.

Introduction to Big Data and Hadoop

# Spark Application Life Cycle  
   
In this lesson, we'll track the various phases a Spark application goes through,  from the time submitted to when it's marked completed.


1. As a user, the first step is to submit the Spark job to the cluster. Usually, this involves the user running the __spark-submit__ command in a terminal window. The command spawns a process that talks to the cluster manager. If YARN is used as the cluster management software then the client process connects to the Resource Manager (RM) daemon. If the job is accepted, the RM will create the Spark driver process on one of the machines in the cluster.

2. Once the driver process starts running, it executes the user code. The code must establish a __SparkSession__ which in sets-up the _Spark cluster_. The driver process and the executor processes are collectively referred to as the Spark cluster. We'll study SparkSession in depth in upcoming lessons. For now, know that the SparkSession is a unified single point of entry to interact with underlying Spark functionality. It allows programming Spark with DataFrame and Dataset APIs. The SparkSession talks to the cluster manager daemon, in our case the RM, to launch Spark executor processes on worker nodes.

3. The RM (cluster manager) will launch Spark executor processes on nodes across the cluster and return the location of executor processes to the driver process. The Spark cluster is setup at this point and the driver can communicate directly with the executor processes.

4. Once the Spark cluster is ready, the driver assigns tasks to executor processes and job execution begins. Data may be moved around and executors report on their status to the driver.

5. The driver exits when the Spark job completes and the cluster manager shuts down the executor processes on behalf of the driver. The cluster manager can be queried for the success or failure of the job.

This lesson talks about the various stages of a Spark application.

Spark Application Life Cycle

Hadoop

YARN

Map Reduce

HDFS

Spark

Input & Output Formats

Misc

Quiz

Reference: Replication

Reference: Partitioning

Reference: Transactions

Reference: Issues in Distributed Systems

Spark Application Life Cycle

Spark Application Life Cycle