Least Squares in Sampling Complexity and Statistical Learning

Data gathering is a constant in human history with ever increasing amounts in quantity and dimensionality. To get a feel for the data, make it interpretable, or find underlying laws it is necessary to fit a function to the finite and possibly noisy data. In this thesis we focus on a method achieving this, namely least squares approximation. Its discovery dates back to around 1800 and it has since then proven to be an indispensable tool which is efficient and has the capability to achieve optimal error when used right. Crucial for the least squares method are the ansatz functions and the sampling points. To discuss them, we gather tools from probability theory, frame subsampling, and L2-Marcinkiewicz-Zygmund inequalities. With that we give results in the worst-case or minmax setting, when a set of points is sought for approximating a class of functions, which we model as a generic reproducing kernel Hilbert space. Further, we give error bounds in the statistical learning setting for approximating individual functions from possibly noisy samples. Here, we include the covariate-shift setting as a subfield of transfer learning. In a natural way a parameter choice question arises for balancing over- and underfitting effect. We tackle this by using the cross-validation score, for which we show a fast way of computing as well as prove the goodness thereof.