Understanding Multi-Layer Perceptrons

Multi-layer perceptrons are possibly one of the most visually illustrated neural networks. Yet most of them lack a few fundamental explanations. Since MLPs are the foundation of deep learning, this section provides a clearer perspective.

MLP architecture

A typical visual representation of an MLP is shown in the illustration above where:

• $\text{X}_\text{1} \text{-} \text{X}_\text{3}$ on the left represent the inputs.
• The middle nodes represent the hidden layers.
• They layer on the right is the output.

This high-level representation shows the feed-forward nature of the network. In a feed-forward network, information between layers flows in only the forward direction. Information (features) learned at a layer is not shared with any prior layer.

The abstracted network shown in the illustration above is unwrapped to its elements in the illustration below.

MLP workflow

The journey through an MLP involves a series of meticulously orchestrated steps, each contributing to the network’s ability to learn and make predictions. Each element, its interactions, and its implementation in the context of TensorFlow are explained step-by-step as follows:

1. Data introduction: The process starts with a dataset. Suppose a dataset is shown at the top left, $X_{n×p}$, with $n$ samples and $p$ features.
2. Batch selection: The model ingests a randomly selected batch during training. The batch contains random samples (rows) from $X$ unless otherwise mentioned. The batch size is denoted as $n_b$ here.
3. Independent processing: By default, the samples in a batch are processed independently. Their sequence is, therefore, not necessary.
4. Entering the input layer: The input batch enters the network through an input layer. Each node in the input layer corresponds to a sample feature. Explicitly defining the input layer is optional, but it is done here for clarity.

5. Navigating hidden layers: The input layer is followed by a stack of hidden layers till the last (output) layer. These layers perform the “complex” interconnected nonlinear operations. Although perceived as complex, the underlying operations are rather simple arithmetic computations.

6. Node functionality: A hidden layer is a stack of computing nodes. Each node extracts a feature from the input. For example, in the sheet-break problem, a node at a hidden layer might determine whether the rotations between two specific rollers are out of sync or not. A node can, therefore, be imagined as solving one arbitrary subproblem.

7. Feature mapping: The stack of output coming from a layer’s nodes is called a feature map or representation. The size of the feature map, also equal to the number of nodes, is called the layer size.

8. Layer-to-layer transmission: Intuitively, this feature map has results of various subproblems solved at each node. They provide predictive information for the next layer until the output layer to predict the response.

9. Perceptron basics: Mathematically, a node is a perceptron made of weights and bias parameters. The weights at a node are denoted with a vector $w$ and a bias $b$.

10. Processing layer inputs: All the input sample features go to a node. The input to the first hidden layer is the input data features $x = \{x_{1}, . . . , x_{p}\}$. For any intermediate layer, it’s the output (feature map) of the previous layer, denoted as $z = \{z_{1},...,z_{m}\}$, where $m$ is the size of the prior layer.

11. Feature extraction logic: Consider a hidden layer $l$ of size $m_l$ in the illustration. A node $j$ in the layer $l$ performs a feature extraction with a dot product between the input feature map $z^{(l−1)}$ ...