Enriching the Basic DataFrame Program

Adding design into the mix

Documentation tends to be a reference, manual, and reminder of an application’s design to consult back and forth. It can also serve as a fundamental blueprint for a project being built, illustrating the landscape and interaction between its parts or components.

As we enrich our application in this lesson by adding a Database into the mix and applying a transformation to the data through the DataFrame API, it is helpful to illustrate how the big picture looks for our application’s changes and, more importantly, how Spark fits into it.

During the duration of this course, we’ll try to enforce this practice, especially when our main batch application increases in its complexity.

The application’s model

Because our application is not complex, the following diagram mixes both architectural and flow perspectives into one:

Get hands-on with 1200+ tech skills courses.