Neural Network-Related Operations
Explore essential neural network operations within TensorFlow including nonlinear activations like sigmoid and ReLU, convolution for feature extraction, pooling for data reduction, and defining loss functions such as mean squared error and cross-entropy. This lesson equips you with foundational knowledge to implement and understand neural network computations critical for deep learning applications.
We'll cover the following...
Now, let’s look at several useful neural network-related operations. The operations we’ll discuss here range from simple element-wise transformations (that is, activations) to computing partial derivatives of a function with respect to a set of parameters. We will also implement a simple neural network as an exercise.
Nonlinear activations used by neural networks
Nonlinear activations enable neural networks to perform well at numerous tasks. Typically, there’s a nonlinear activation transformation (that is, activation layer) after each layer output in a neural network (except for the last layer). A nonlinear transformation helps a neural network to learn various nonlinear patterns that are present in data. This is very useful for complex real-world problems, where data often has more complex nonlinear patterns. If not for the nonlinear activations between layers, a deep neural network would be a bunch of linear layers stacked on top of each other. Also, a set of linear layers can essentially be compressed into a single bigger linear layer.
In conclusion, if not for the nonlinear activations, we couldn’t create a neural network with more than one layer.
Let’s observe the importance of nonlinear activation through an example. First, recall the computation for the neural networks we saw in the sigmoid example. If we disregard , it will be this:
Assume a three-layer neural network (having , , and as layer weights) where each layer does the preceding computation; we can summarize the full computation as follows:
...