Diverse Autoencoders and Their Applications

There are several types of autoencoders. The table below summarizes the properties of the most common autoencoders. The rest of this lesson briefly describes them along with their applications.

Undercomplete autoencoders

An undercomplete autoencoder has a smaller encoding dimension than the input. A simple example of such an autoencoder is

$$X_{·×p} → \underbrace{f(X_{·×p}W^{(e)}_{p×k})}_{encoder} → Z_{·×k} → \underbrace{g(Z_{·×k}W^{(d)}_{k×p})}_{decoder} → \hat{X}_{·×p}.$$

Here, the input $X$ and the encoding $Z$ are $p$- and $k$-dimensional, respectively, and $k < p$.

In learning a smaller representation, an undercomplete autoencoder gathers the most salient features of the data. The learning process is simply minimizing a loss function,

$$L(x,g(f(xW^{(e)})W^{(d)}))$$

where $L$ is a loss function, for example, mean squared error, that penalizes dissimilarity between $x$ and $\hat{x} = g(f(xW^{(e)})W^{(d)})$. Undercomplete autoencoders are more common. Perhaps because it has roots in PCAPrincipal component analysis. A linearly activated (single or multilayer) undercomplete autoencoder reduces to,

$$X_{·×p} → X_{·×p}W^{(e)}_{p×k} → Z_{·×k} → Z_{·×k}W^{(d)}_{k×p} → \hat{X}_{·×p},$$

which is the same as PCA if $W^{(e)}$ ...