Rescale the Target Output

The need to rescale the output

We now need to think about the neural network’s outputs. We saw earlier that the outputs should match the range of values that the activation function can push out. The logistic function we’re using can’t push out numbers like $-2.0$ or $255$. The range is $0.0–1.0$, and in fact, we can’t reach $0.0$ or $1.0$ because the logistic function can only approach these extremes and will never actually get there. So, it looks like we’ll have to scale our target values when training.

What should the output even be? Should it be an image of the answer? That would mean we’d have $28\times28 = 784$ output nodes.

If we take a step back and think about what we’re asking the neural network to do, we realize we’re asking it to classify the image and assign the correct label. That label is one of $10$ numbers, or $0–9$. That means the network should be able to have an output layer of $10$ nodes, one for each of the possible answers, or labels. If the answer is $0$, the first output layer node would fire and the rest should be silent. If the answer is $9$, the last output layer node would fire and the rest would be silent.

Examples

The following table illustrates this scheme with some example outputs: