IntelliJ: Debug and Inspect Spark Execution
Explore how to debug Apache Spark applications with IntelliJ IDEA by running and inspecting Spark Java code locally. Understand setting breakpoints, evaluating expressions, and analyzing Spark object states during execution. Gain practical skills to troubleshoot and optimize Spark applications in a real-time development environment.
We'll cover the following...
Working locally
One advantageous property of using Spark as a Java library is that the code can be run and, more interestingly, debugged in real-time.
Debugging in particular is very valuable because it allows developers to conduct local live-testing, and to understand, through inspection of Spark objects, the nature of Spark-based applications.
This lesson demonstrates how to debug a Spark-based project, so let’s start right away.
Note: The lesson illustrates the necessary actions in IntelliJ IDEA IDE. Feel free to investigate plenty of options on the Web for debugging in Eclipse or NetBeans, if that happens to be the preferred “flavor” for a development environment.
Debugging the basic example
Let’s briefly recall the lesson “Running The First Spark Program."
To load the project from the code widget into IntelliJ IDEA IDE locally, the ...