In the polynomial example, perhaps a very high degree polynomial is able to approximate the cubic spline interpolator, which has much lower variance than the Lagrange polynomial (the interpolating polynomial of minimum degree). Cubic splines are provably the lowest variance interpolating piecewise polynomials.
It may depend on how one is choosing a solution in the underdetermined case (degree > data points - 1). Will L2 regularization give something close to the cubic spline?
In the polynomial example, perhaps a very high degree polynomial is able to approximate the cubic spline interpolator, which has much lower variance than the Lagrange polynomial (the interpolating polynomial of minimum degree). Cubic splines are provably the lowest variance interpolating piecewise polynomials.
It may depend on how one is choosing a solution in the underdetermined case (degree > data points - 1). Will L2 regularization give something close to the cubic spline?
How would you explain it in the polynomial case?