Activation Function

Learn about different activation functions: what they are and how a change in network architecture with a different activation function may increase the network performance.

Logistic function

The s-shaped logistic function was used in the early days of neural networks because its shape seemed to match what we thought was happening in animals as a threshold for signals passing between neurons, and also because it is mathematically convenient for calculating gradients.

However, it does have some weaknesses. The main one is that for large values, the gradients get very small, and can effectively disappear. This means that training neural networks, which depend on gradients to correct the link weights, can start to fail when this kind of saturation happens.

Get hands-on with 1200+ tech skills courses.