Foundational Aspects of Theory of Statistical Function Estimation and Pattern Recognition

This paper provides a gentle introduction to the foundational ideas, concepts and results in the field of science dedicated to the theory of statistical function estimation and pattern recognition. The so-called VC Theory of Vapnik and Chervonenkis is introduced and explored gradually. The emphasis is placed on helping the reader appreciate the importance of the extension of the classical law of large numbers to function spaces, and the key role that "new" concepts such as Empirical Risk Minimization (ERM) principle, ERM consistency, VC dimension, and complexity control play in constructing algorithms that yield function estimators with optimal properties. As much as possible, each key concept is introduced via a tangible example, with the hope of helping the reader grasp the essential core of the foundational concept under exploration.

Statistical Learning Theory, Law of Large Numbers, Consistency, VC Theory, Regularization, Complexity Control, Bounds on generalization, Generalization