Trusted answers to developer questions

Sarvech Qadir

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

**Regularization** is a form of regression. In regularization, the estimated coefficients in regression equations are constrained to zero. This is one of the major techniques used to avoid overfitting.

Regularization is a method that can be utilized for any given training set. It reduces errors by fitting the function appropriate to the training set. As a result, this method avoids overfitting.

The commonly used regularization techniques are:

**Lasso Regression (L1 regularization)**calculates the absolute value of the coefficients’ magnitude and adds it to the loss function.

General equation looks like:

```
y = β0 + β1x1 + β2x2 + ··· βkxk + λ(slope)²
```

Loss function: $||w||_1 = |w_1| + |w_2| + ... |w_N|$

**Ridge Regression (L2 regularization)**calculates the square value of the coefficients’ magnitude and adds it to the loss function. The general equation looks like:

```
y = β0 + β1x1 + β2x2 + ··· βkxk + λ|slope|
```

Loss function: $||w||_2 = (|w_1|^2 + |w_2|^2 + ... |w_N|^2)^{1/2}$

Output function:

$y^{hat} = w_1x_1 + w_2x_2 + ... w_Nx_N + b$

We define the loss function in logistic regression as:

```
L(y_hat,y) = y log y_hat + (1 - y)log(1 - y_hat)
```

Loss function with no regularization:

```
L = y log (wx + b) + (1 - y)log(1 - (wx + b))
```

Let’s say the data overfits the above function.

Loss function with L1 regularization:

```
L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w||
```

Loss function with L2 regularization:

```
L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w|| ^ 2
```

Note that lambda is a hyperparameter known as the regularization constant – it is greater than zero.

RELATED TAGS

general

CONTRIBUTOR

Sarvech Qadir

Copyright ©2022 Educative, Inc. All rights reserved

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Keep Exploring

Related Courses