[np-ml] Ridge Regression

合集 - numpy-ml(2)

1.[np-ml] Ridge Regression2024-09-14

2.[np-ml] Linear Regression2024-09-14

收起

算法描述

Ridge regression uses the same simple linear regression model but adds an additional penalty on the $L2$-norm of the coefficients to the loss function. This is sometimes known as Tikhonov regularization.

In particular, the ridge model is the same as the ordinary Linear Squares model:

$$\mathbf{y}=\mathbf{bX}+ϵ$$

where $\epsilon\sim\mathcal{N}(0, \sigma^{2}), except now the error for the model is calculated as:

$$\mathcal{L}=\Vert \mathbf{y}-\mathbf{b}\mathbf{X}{\Vert }_2^2+\alpha \Vert \mathbf{b}{\Vert }_2^2$$

The MLE for the model parameter $\mathbf{b}$ can be computed in closed form via the adjusted normal equation:

$${\hat{\mathcal{b}}}_{\text{ridge}}=({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}+\alpha \mathbf{I}{)}^{-1}{\mathbf{X}}^{\mathrm{\top }}\mathbf{y}$$

where $({\mathbf{X}}^{\mathrm{\top }}\mathbf{X}+\alpha \mathbf{I}{)}^{-1}$ is the pseudo-inverse/Moore-Penrose inverse adjusted for $L2$ penalty on the model coefficients.

代码实现

import numpy as np
def fit(X, y, fit_intercept=True, alpha=1):
    if fit_intercept:
        X = np.c_[np.ones(X.shape[0]), X]
    
    A = alpha * np.eye(X.shape[1])
    pseudo_inverse = np.linalg.inv(X.T @ X + A) @ X.T
    beta = pseudo_inverse @ y
    return beta

参考资料

code
doc-overview
doc