Overfitting happens with excessive power, stochastic/deterministic noise, and limited data.

1. Regularized Hypothesis Set

在实际训练过程中，复杂的模型容易产生过拟合现象，最终导致得到的训练模型的$E_{out}$变得非常大。一种很自然的想法就是给模型加上一些附加限制，用以避免过拟合现象，同时使得模型尽可能的简单化！

Regression with Looser Constraint：

Regression with Softer Constraint：

2. Weight Decay Regularization

The Lagrange Multiplier

(对于上图关于The Lagrange Multiplier的直观解释，让我印象深刻！)

Some Detail: Legendre Polynomials (正则化过程中，可以选择正交的基函数)

3. Regularization and VC Theory

4. General Regularizers

Regularizers: constraint in the "direction" of target function.

The Optimal $\lambda$