Principal Component Analysis
Explore Principal Component Analysis to understand and apply dimensionality reduction techniques that simplify complex data sets by minimizing information loss. This lesson covers the curse of dimensionality, PCA properties, eigenvalue decomposition, and a practical PCA implementation using the Iris dataset.
We'll cover the following...
In the previous chapter, we explored ensemble learning, where combining multiple models improved predictive performance. While these models are powerful, working with high-dimensional feature spaces can still be challenging due to computational costs, data sparsity, and the risk of overfitting.
Now, it’s time to simplify the data without losing essential information. This is where dimensionality reduction comes in. Before we dive into techniques like PCA, let’s first understand the curse of dimensionality, which refers to the challenges that arise when dealing with high-dimensional datasets and why reducing dimensions is so valuable.
Curse of dimensionality
Curse of dimensionality in machine learning refers to the challenges and computational complexities that arise when dealing with a large number of features (high-dimensional data/high-dimensional feature space). As the number of features or dimensions increases, the amount of data needed to maintain reliable and meaningful patterns also increases, often leading to increased data and computational demands and the risk of overfitting.
Example
Consider a product recommendation system where each product is described by multiple features such as price, size, color, brand, and so on. As the number of features increases, possible combinations grow exponentially, making it harder to find meaningful relationships between products and user preferences. This high-dimensional data can lead to sparse data points, which makes accurate predictions more challenging and requires more data to avoid unreliable results, hence, illustrating the curse of dimensionality.
It seems desirable to reduce the number of features by maintaining the information. Does the term “compression” ring a bell?
Dimensionality reduction
Dimensionality reduction involves decreasing the number of features and is achieved by either selecting the most significant ones or by transforming them into a smaller set of new features. Not all dimensionality reduction methods aim to maintain information (to reconstruct or decompress). Different objectives can be defined in this regard.
PCA
Principal Component Analysis (PCA) is a dimensionality reduction technique that identifies key patterns and relationships within data by projecting it onto a lower-dimensional space while preserving as much variance (spread or information) as possible.
We first need to understand the dimensions to understand PCA. Imagine you’re in a video game where you can move forward, backward, left, and right. These are two dimensions. Now, imagine you can also fly up or dig down. ...
Properties of PCA
PCA operates by finding a new set of orthogonal (perpendicular) axes, called principal components (, that are oriented in the directions where the data is most spread out.
To explain the essential properties of PCA, let’s take an example of data points in ...