Trusted answers to developer questions
Trusted Answers to Developer Questions

Related Tags

general

What is regularized regression?

Sarvech Qadir

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Regularization is a form of regression. In regularization, the estimated coefficients in regression equations are constrained to zero. This is one of the major techniques used to avoid overfitting.

Regularization is a method that can be utilized for any given training set. It reduces errors by fitting the function appropriate to the training set. As a result, this method avoids overfitting.

Regularization techniques

The commonly used regularization techniques are:

  1. Lasso Regression (L1 regularization) calculates the absolute value of the coefficients’ magnitude and adds it to the loss function.

General equation looks like:

y = β0 + β1x1 + β2x2 + ··· βkxk + λ(slope)²

Loss function: w1=w1+w2+...wN||w||_1 = |w_1| + |w_2| + ... |w_N|

  1. Ridge Regression (L2 regularization) calculates the square value of the coefficients’ magnitude and adds it to the loss function. The general equation looks like:
y = β0 + β1x1 + β2x2 + ··· βkxk + λ|slope|

Loss function: w2=(w12+w22+...wN2)1/2||w||_2 = (|w_1|^2 + |w_2|^2 + ... |w_N|^2)^{1/2}

Output function:

yhat=w1x1+w2x2+...wNxN+by^{hat} = w_1x_1 + w_2x_2 + ... w_Nx_N + b

Loss function in regularized regression

We define the loss function in logistic regression as:

L(y_hat,y) = y log y_hat + (1 - y)log(1 - y_hat)

Loss function with no regularization:

L = y log (wx + b) + (1 - y)log(1 - (wx + b))

Let’s say the data overfits the above function.

Loss function with L1 regularization:

L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w||

Loss function with L2 regularization:

L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w|| ^ 2   

Note that lambda is a hyperparameter known as the regularization constant – it is greater than zero.

RELATED TAGS

general

CONTRIBUTOR

Sarvech Qadir
Copyright ©2022 Educative, Inc. All rights reserved

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Keep Exploring