Introduction to Logistic Regression

Learn about the sigmoid function, logistic regression, and its optimization via BCE loss.

Logistic Regression

Logistic regression is a discriminative model widely used for classification tasks. The term “logistic” in logistic regression refers to the utilization of the logistic function. Consider a binary classification problem: the target variable yiy_i can take on values in {0,1}\{0, 1\}. One way to model the probability distribution of the target label yiy_i being equal to 1 given the feature vector ϕ(xi)\phi(\bold{x}_i) is by employing a logistic function defined as:

y^i=p(yi=1ϕ(xi))=11+ewTϕ(xi)\hat{y}_i=p(y_i=1|\phi(\bold{x}_i))=\frac{1}{1+e^{-\bold w^T \phi(\bold{x}_i)}}

The complementary probability p(yi=0ϕ(xi))p(y_i=0|\phi(\bold x_i)) can be obtained as:

p(yi=0ϕ(xi))=1p(yi=1ϕ(xi))p(y_i=0|\phi(\bold{x}_i))= 1-p(y_i=1|\phi(\bold{x}_i))

Sigmoid function

A logistic function is also known as sigmoid, typically denoted by σ(z)\sigma(z).

σ(z)=11+ez\sigma(z)=\frac{1}{1+e^{-z}}

Get hands-on with 1200+ tech skills courses.