Interpreting Spark Logs

Get introduced to Spark application logs and the key parts to understanding them.

Logging and Spark

Whenever Spark is being used in a Java (or Spring) application, the logs produced by it are most likely a mix of the log lines included by the developers in the Java application, plus the logs produced by the Spark libraries themselves.

This separation occurs because Spark libraries work alongside logging libraries such as (and at least currently in version 3.X) org.slf4j and log4j logging frameworks (with the interface being the former, and the implementation the latter). We purposely excluded this library in the batch template Maven project within the pom.xml file to avoid compatibility issues with Spring Boot’s logging framework used:

Get hands-on with 1200+ tech skills courses.