Multiclass Formulation

Multiclass extension

The logistic regression model offers two significant advantages: the ability to learn probabilities and the natural extension to handle multiple classes. Let’s explore how we can extend a binary classification model to a multiclassification model with $c$ classes. This technique is also called one-vs-all (one-vs-rest).

Algorithm

For each class $j$ in the dataset:

1. Set the labels of class $j$ to 1, indicating positive instances.

2. Set the labels of all other classes to 0, representing negative instances.

3. Apply logistic regression on the modified dataset, treating it as a binary classification problem with class $j$ as the positive class (1) and all other classes as the negative class (0). Save the corresponding model parameters $\boldsymbol{\bold w_j}$.

4. Predict the probability $\hat{y}_{tj} = \sigma(\bold w^T_j\phi(\bold x_t))$, where $\bold{x}_t$ is the input vector of the test instance.

5. Rescale all class predictions $\hat{y}_{tj}$ by dividing each by the sum of all predictions across all classes $\sum_{j=1}^c \hat{y}_{tj}$. This step ensures that the probabilities sum up to 1, providing normalized probabilities.

6. Finally, select the class with the maximum rescaled prediction as the predicted class for the test instance $\bold{x}_t$.