Linear discriminant analysis

Linear discriminant analysis (LDA) is a statistical technique for dimensionality reduction. We use this technique if the features are continuous and the target value is categorical.

LDA aims to maximize the distance between the means of different classes while minimizing the variance within each class.

Example

Suppose we have a dataset with two classes: Class A and Class B with each class having multiple samples. Linear discriminant analysis (LDA) aims to find a linear discriminant or unit vector to project our data to maximize the gap between different classes while reducing the variation within each class. This helps us better distinguish between the classes and make more accurate predictions.

Procedure

We want to find a special direction (eigenvector or axis) that would allow us to project our data points to maximize the separation between different classes' average values while minimizing the variation within each class.

Let's consider a simple example with two classes: "Apples" and "Oranges." Each class has two features: "Sweetness" and "Sourness." We have the following data points:

Step 1: Compute the mean vector of each class:

Step 2: Calculate the distance between the means. This distance will be the separability between the datasets:

Step 3: Now calculate the scatter (variance) of each class within themselves:

For apples:

For oranges:

Step 4: Now, we need to keep on finding an axis (or vector) upon which the data points, after getting projected, will have a maximum value of:

Note: Learn about the implementation of LDA in python.

Conclusion

By projecting the data onto the LDA axis, we can achieve better separability and discrimination between classes, making it easier to classify new data points accurately. However, it's important to note that LDA assumes the data follows a normal distribution and that the classes have identical covariance matrices. In cases where these assumptions are violated, the performance of LDA may not be optimal.

Note: Read about principle component analysis.