Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Resampling and Class Weighting

Explore methods for managing imbalanced datasets in classification tasks by applying resampling and class weighting. Understand the risks of data leakage, how to properly split datasets, and methods like undersampling, oversampling, SMOTE, and class weighting. Learn to evaluate models reliably using untouched validation data and interpret the impact of these techniques on recall and precision.

We'll cover the following...

Oversampling a rare-event dataset before a train/test split can produce a validation score that looks excellent while the model has already seen duplicates of its test examples. The score is wrong because evaluation data must represent unseen cases. After diagnosing imbalance with recall and average precision, we can help a classifier find positives in two different ways without contaminating that evaluation.

Resampling changes the class balance of the training data. Class weighting leaves the training rows unchanged but makes mistakes on the minority class count more during fitting. Neither option is automatically better. The right choice depends on the data, the model, and whether generated or duplicated examples would distort the problem.

Change the training distribution

Random undersampling removes some majority-class rows. If a fraud training set contains 90 legitimate transactions and 10 fraud cases, undersampling can retain all 10 fraud cases while dropping many legitimate rows. The resulting training set is smaller, so fitting becomes faster, but the classifier loses information about legitimate transaction patterns.

Random oversampling duplicates minority-class rows. The same 10 fraud cases may appear repeatedly until the training classes have similar counts. Oversampling preserves every majority-class row, but repetition can make a flexible model fit peculiarities of the few original positive examples.

The counts and predicted positive rates below show how changing the training distribution can change a logistic regression model’s decisions on the same untouched test inputs.

Python
from collections import Counter
import numpy as np
from imblearn.over_sampling import RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
from sklearn.linear_model import LogisticRegression
def make_tiny_imbalanced_dataset(random_state: int = 42):
    rng = np.random.default_rng(random_state)
    n_train_neg, n_train_pos = 90, 10
    n_test_neg, n_test_pos = 90, 10
    # Two numeric features where positives are shifted upward.
    X_train_neg = rng.normal(loc=0.0, scale=1.0, size=(n_train_neg, 2))
    X_train_pos = rng.normal(loc=1.5, scale=1.0, size=(n_train_pos, 2))
    X_train = np.vstack([X_train_neg, X_train_pos])
    y_train = np.array([0] * n_train_neg + [1] * n_train_pos)
    X_test_neg = rng.normal(loc=0.0, scale=1.0, size=(n_test_neg, 2))
    X_test_pos = rng.normal(loc=1.5, scale=1.0, size=(n_test_pos, 2))
    X_test = np.vstack([X_test_neg, X_test_pos])
    y_test = np.array([0] * n_test_neg + [1] * n_test_pos)
    # Shuffle each split so class blocks aren't contiguous.
    train_perm = rng.permutation(len(y_train))
    test_perm = rng.permutation(len(y_test))
    return (
        X_train[train_perm],
        y_train[train_perm],
        X_test[test_perm],
        y_test[test_perm],
    )
def positive_rate(model: LogisticRegression, X: np.ndarray) -> float:
    y_pred = model.predict(X)
    return float(np.mean(y_pred == 1))
def main() -> None:
    X_train, y_train, X_test, y_test = make_tiny_imbalanced_dataset(random_state=42)
    original_counts = Counter(y_train)
    print("Original training class counts:", {int(k): v for k, v in sorted(original_counts.items())})
    rus = RandomUnderSampler(random_state=42)
    X_train_under, y_train_under = rus.fit_resample(X_train, y_train)
    under_counts = Counter(y_train_under)
    print("After random undersampling:", {int(k): v for k, v in sorted(under_counts.items())})
    ros = RandomOverSampler(random_state=42)
    X_train_over, y_train_over = ros.fit_resample(X_train, y_train)
    over_counts = Counter(y_train_over)
    print("After random oversampling:", {int(k): v for k, v in sorted(over_counts.items())})
    # Keep the test set fixed and untouched so we can compare decisions fairly.
    baseline_model = LogisticRegression(solver="liblinear", random_state=42)
    baseline_model.fit(X_train, y_train)
    under_model = LogisticRegression(solver="liblinear", random_state=42)
    under_model.fit(X_train_under, y_train_under)
    over_model = LogisticRegression(solver="liblinear", random_state=42)
    over_model.fit(X_train_over, y_train_over)
    print()
    print(f"Baseline model predicted positive rate on test: {positive_rate(baseline_model, X_test):.3f}")
    print(f"Undersampled model predicted positive rate on test: {positive_rate(under_model, X_test):.3f}")
    print(f"Oversampled model predicted positive rate on test: {positive_rate(over_model, X_test):.3f}")
if __name__ == "__main__":
    main()

  • Lines 4–6: The code imports RandomOverSampler and RandomUnderSampler from imbalanced-learn and LogisticRegression from scikit-learn.

  • Lines 12–24: The train and ...