Supervised Generative Models

1D Gaussian example

Classification with generative models has been used for some time. We’ll be discussing an example here which is related to a method called linear discriminant analysis that goes back to a paper by Fisher in 1936. This is the same Fisher who collected the Iris dataset. We will start by outlining the idea in a 1-dimensional Gaussian model, before deriving the more general case with more attributes and classes.

An example of the distribution of an attribute $x$ for two classes is shown in the figure below. As can be seen, these classes are not fully discriminated by this attribute value, because the attribute values are overlapping. Thus, a good note to remember is that $100$ percent classification is not always possible. However, we can still predict with some high confidence in some cases. A good choice, and indeed the best possible choice, is to predict the blue class on the left for attribute values less than $1$ and the red class on the right for attribute values above $1$.