A Simple RNN Cell
Explore the structure and function of a simple recurrent neural network (RNN) cell, which processes sequences timestep by timestep, maintaining memory across time. Understand sequence unrolling and how backpropagation through time allows training of these models, including challenges like linear complexity and practical training tips.
As we already know, convolutional layers are specialized for processing grid-structured data (i.e., images). On the contrary, recurrent layers are designed for processing sequences.
To distinguish Recurrent NN (RNNs) from fully-connected layers, we call the non-recurrent networks feedforward NN.
The smallest computational unit in recurrent networks is the cell. The cell is the basic unit in recurrent networks.
Recurrent cells are NN for processing sequential data. They are usually small.
Recurrent models help us deal with time-varying signals, so always have the notion of time and timesteps in the back of your mind.
A minimal recurrent cell: sequence unrolling
One can create a minimal recurrent unit by connecting the current timesteps’ output to the input of the next timestep!
This is the core recent principle and is called sequence unrolling. Note that the unrolling can happen in any dimension, but we usually refer to time.
But why do we choose the time dimension?
We choose to model the time dimension with RNNs because we want to ...