Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

From Linear to Logistic: The Sigmoid and Log-Loss

Explore how logistic regression transforms linear predictions into probabilities using the sigmoid function and learns through log-loss to penalize incorrect predictions. Understand the importance of probabilistic outputs in classification tasks like fraud detection and loan default prediction, and how this approach improves model reliability over simple linear regression.

We'll cover the following...

A fraud screening model must return a risk score for a payment, not an unbounded estimate. For example, a payment worth $20 might receive a fraud probability near 0.05, while a $4,000 payment might receive a higher probability. The final decision can use that score, but the score itself must behave like a probability.

A linear regression model can learn a line from transaction amount to the binary fraud label 0 or 1. The wrong intuition is that a line constrained by examples labeled 0 and 1 will also stay between 0 and 1 for new payment amounts. A straight line has no such boundary. It can return a negative number for a small amount or a number greater than 1 for a large amount.

Python
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
def main() -> None:
    transactions = pd.DataFrame(
        {
            "amount": [5, 12, 18, 25, 40, 60, 85, 120, 200, 350, 600, 900],
            "is_fraud": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
        }
    )
    X = transactions[["amount"]]
    y = transactions["is_fraud"]
    model = LinearRegression()
    model.fit(X, y)
    test_amounts = pd.DataFrame({"amount": [10, 20, 50, 200, 4000, 5000]})
    raw_predictions = model.predict(test_amounts[["amount"]])
    results = test_amounts.copy()
    results["linear_prediction"] = raw_predictions
    results["outside_[0,1]"] = (results["linear_prediction"] < 0) | (results["linear_prediction"] > 1)
    pd.set_option("display.max_columns", None)
    pd.set_option("display.width", 120)
    pd.set_option("display.float_format", lambda v: f"{v:0.4f}")
    print(results.to_string(index=False))
if __name__ == "__main__":
    main()

  • Lines 7–12: The small in-memory transaction table uses a binary is_fraud label, but fitting LinearRegression on it still learns an unconstrained line rather than a probability model.

  • Lines 20–25: Predicting on extreme amounts (like 4000 and 5000) makes the unbounded nature of a linear model visible; the outside_[0,1] flag highlights any raw predictions that fall below 0 or above 1, which is why these outputs cannot be interpreted as probabilities.

  • Lines 27–31: Display formatting is set so the printed table is compact and the numeric issue is easy to spot (e.g., values like 5.7420 and 7.1658 stand out clearly).

This code produces raw predictions outside the valid probability range. A prediction such as 5.74 cannot mean a 574% chance of fraud, and a negative prediction, which a line can also produce for other inputs, could not be a probability either. For a binary outcome, where the label is either 0 for not fraud or 1 for fraud, the model needs an output strictly between 0 and 1.

Modeling requirement: A fraud risk score can act as a probability only when it falls between 0 and 1.

Mapping scores to probabilities

Because a linear score can take any real value, logistic regression passes that score through the sigmoid function, a function that maps every real-valued input into the open interval from 0 to 1. The model can still use a linear combination of features internally, but the sigmoid converts the resulting score into a valid fraud probability.

For a score zz, the sigmoid is:

When zz ...