Trusted answers to developer questions
Trusted Answers to Developer Questions

Related Tags

general

# What is regularized regression?

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Regularization is a form of regression. In regularization, the estimated coefficients in regression equations are constrained to zero. This is one of the major techniques used to avoid overfitting.

Regularization is a method that can be utilized for any given training set. It reduces errors by fitting the function appropriate to the training set. As a result, this method avoids overfitting.

## Regularization techniques

The commonly used regularization techniques are:

1. Lasso Regression (L1 regularization) calculates the absolute value of the coefficients’ magnitude and adds it to the loss function.

General equation looks like:

y = β0 + β1x1 + β2x2 + ··· βkxk + λ(slope)²


Loss function: $||w||_1 = |w_1| + |w_2| + ... |w_N|$

1. Ridge Regression (L2 regularization) calculates the square value of the coefficients’ magnitude and adds it to the loss function. The general equation looks like:
y = β0 + β1x1 + β2x2 + ··· βkxk + λ|slope|


Loss function: $||w||_2 = (|w_1|^2 + |w_2|^2 + ... |w_N|^2)^{1/2}$

Output function:

$y^{hat} = w_1x_1 + w_2x_2 + ... w_Nx_N + b$

## Loss function in regularized regression

We define the loss function in logistic regression as:

L(y_hat,y) = y log y_hat + (1 - y)log(1 - y_hat)


Loss function with no regularization:

L = y log (wx + b) + (1 - y)log(1 - (wx + b))


Let’s say the data overfits the above function.

Loss function with L1 regularization:

L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w||


Loss function with L2 regularization:

L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w|| ^ 2


Note that lambda is a hyperparameter known as the regularization constant – it is greater than zero.

RELATED TAGS

general

CONTRIBUTOR 