Search⌘ K
AI Features
Home
Courses
Mastering Big Data with Apache Spark and Java

Interpreting Spark Logs

Explore how to interpret Apache Spark logs within Java applications to track processing jobs, stages, and resources. Understand the role of SparkContext, SparkUI, and key log entries to monitor Spark's performance. Learn to connect log lines with code actions to troubleshoot and optimize big data Java applications using Spark.

We'll cover the following...

Logging and Spark

Whenever Spark is being used in a Java (or Spring) application, the logs produced by it are most likely a mix of the log lines included by the developers in the Java application, plus the logs produced by the Spark libraries themselves.

This separation occurs because Spark libraries work alongside logging libraries such as (and at least currently in version 3.X) org.slf4j and log4j logging frameworks (with the interface being the former, and the implementation the latter). We purposely excluded this library in the batch template Maven project within the pom.xml file to avoid compatibility issues with Spring Boot’s logging framework used:

XML
    <!-- Spark -->
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-core_${scala.version}</artifactId>
			<version>${spark.version}</version>
			<exclusions>
				<exclusion>
					<groupId>org.slf4j</groupId>
					<artifactId>slf4j-simple</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

Technical quirks aside, it is easy to spot the Spark logs produced by an application because they are preceded by the fully qualified name (package.ClassName) ...