Machine-learning 5 minutes

What is ridge regression?

A ridge regression is an OLS regression that uses L2-regularization.

The regularized loss function on some dataset is thus:

where is an hyper-parameter that controls the importance of the regularization.

For a complete discussion about the effect of L2-regularization on the parameters of the model, check out our dedicated article: L2-regularization.

Common mistake: it is important to normalize the features before using regularization. Failure to do so will yield incoherent regularization behavior.

Analytical solution

Let be the training-set and note and the corresponding design matrix and output vector.

We can compute the value of the parameter vector that minimizes the regularized loss using differentiation.

 
 

Setting all directional derivatives to , we get: