Search⌘ K
AI Features
Home
Courses
Fundamentals of Machine Learning: A Pythonic Introduction

SVM in Sklearn

Explore how Support Vector Machines extend to multiclass classification using the one-vs-all method in scikit-learn. Learn to train linear SVMs on the Iris dataset, visualize decision boundaries, and make predictions. This lesson helps you understand the multiclass extension of binary SVMs and apply it with hands-on Python examples for effective classification.

We'll cover the following...

Support Vector Machines (SVMs) are naturally designed for binary classification, where the goal is to separate data into two classes using the maximum-margin decision boundary. However, many real-world problems involve more than two classes, which requires extending the binary SVM framework. In this lesson, we explore how SVM handles multiclass classification, the common one-vs-all strategy, and how to implement it using scikit-learn.

Multiclass classification

SVMs are often used for binary classification, where the goal is to separate data into two classes. However, SVMs can also be used for multiclass classification, where the goal is to separate data into more than two classes.

Multiclass SVM
Multiclass SVM

The above illustration shows the multiclass classification of the Iris dataset using SVM.

One-vs-all

Multiclass SVM is an extension of binary SVM, where the SVM is trained to classify data into more than two classes. There are several approaches to training a multiclass SVM, but the most common approach is to use the one-vs-all (OVA) method. In the OVA method, a separate binary SVM is trained for each class, with the data of that class as the positive class and the data of all other classes as the negative class. During testing, to determine the final class for a new data point, the model compares the scores (or output values) generated by all the individual SVMs. The class corresponding to the single SVM that produced the largest score is assigned as the final predicted class.

OVA method in SVM
OVA method in SVM

Note: Both the hard-margin and soft-margin SVM algorithms are inherently binary classification algorithms and perform poorly when directly extended to multiclass classification. The common multiclass extension (One-vs-All) is a heuristic (“hack”) and lacks the probabilistic interpretation found in some other machine learning models.

Algorithm

Here are the steps to train a multiclass SVM using the OVA method:

  1. Collect the data for each class and preprocess it as necessary.

  2. Split the data into training and testing sets. The training set will be used to train the SVM, and the testing set will be used to evaluate its performance.

  3. Train a binary SVM for each class using the training set. For the specific class currently being trained (e.g., Class A), the data belonging to that class (the training class data) is used as the positive class (yi=+1y_i = +1 ...