Limitations of Linear Classifiers

Introduction

The simple predictors and classifiers are the ones that take some input, do some calculations, and produce an answer. Although they’re fairly effective, they’re not complex enough to solve some of the more interesting problems we hope to tackle with neural networks.

Here, we’ll illustrate the limit of a linear classifier with a simple but stark example. Why do we want to do this and not jump straight into discussing neural networks? The reason is that a key design feature of neural networks comes from understanding this limit. So, it’s worth spending a little time on.

Logic functions

We’ll be moving away from garden bugs and looking at boolean logic functions. George Boole was a mathematician and philosopher. His name is associated with simple functions like AND and OR.

Boolean logic functions are like language or thought functions. If we say, “You can only have your pudding if you’ve eaten your vegetables AND you’re still hungry,” we’re using the boolean AND function. The boolean AND is only true if both conditions are true. It’s not true if only one of them is true. So, if we’re hungry, but we haven’t eaten our vegetables, then we can’t have ...