Beyond the Sigmoid

Explore ReLU activation function and discover when to use which activation function.

We'll cover the following

There is no such thing as a perfect replacement for the sigmoid. Different activation functions work well in different circumstances, and researchers keep coming up with the new ones. That being said, one activation function has proven so broadly useful that it’s become a default of sorts. Let’s discuss it in the next section.

Enter the ReLU

The go-to replacement for the sigmoid these days is the rectified linear unit or ReLU. Compared with the sigmoid, the ReLU is surprisingly simple. Here’s a Python implementation of it:

Get hands-on with 1200+ tech skills courses.