Further Investigation on BYY Harmony Learning in Comparison with Typical Learning Theories for Model Selection
(L. Xu)

Statistical learning takes essential roles in neural networks, pattern recognition, data mining, and bioinformatics, etc. To use a model to appropriately fit dependence structures of the underlying world that samples comes, one critical challenge is called model selection, namely selecting an appropriate scale for a chosen model type. The difficulty is that the scale cannot be determined appropriately by the standard modelling theory of seeking a best fitting between the model and samples. Several theories have been proposed for solving this difficulty, including AIC, CAIC, BIC (equivalently MDL), cross-validation, VC-based SRM. These theories have to be implemented via two stages. First, a number of candidates of a same model type in different scales are considered with parameters in each candidate learned for a best fitting to the samples via the maximum likelihood principle. Second, the best is selected among these candidates according a criterion from one of these theories. Not only such a two stage implementation costs very expensively, but also there lack comparative studies on these theories. One aim of this proposal is a systematic comparison via mathematical analysis and empirical studies on these theories and also on the recently proposed BYY harmony learning that is able to make model selection automatically during parameter learning without a two stage implementation. The other aim is further improve the BYY harmony learning in the cases of a small size of samples, in help of data smoothing regularization and also tools developed in studies of AIC, CAIC, CV, etc, which can be used in either practical applications or comparative studies for developing new model selection approaches.