Generalization and Maximum Likelihood from Small Data Sets

Download: PDF.

“Generalization and Maximum Likelihood from Small Data Sets” by W. Byrne. In IEEE-SP Workshop on Neural Networks in Signal Processing, 1993.

Abstract

An often encountered learning problem is maximum likelihood training of exponential models. When the state is only partially specified by the training data, iterative training algorithms are used to produce a sequence of models that assign increasing likelihood to the training data. Although the performance as measured on the training set continues to improve as the algorithms progress, performance on related data sets may eventually begin to deteriorate. The cause of this behavior can be seen when the training problem is stated in the Alternating Minimization framework. A modified maximum likelihood training criterion is suggested to counter this behavior. It leads to a simple modification of the learning algorithms which relates generalization to learning speed. Training Boltzmann Machines and Hidden Markov Models is discussed under this modified criterion.

Download: PDF.

BibTeX entry:

@inproceedings{byrne93:_gener_,
   author = {W. Byrne},
   title = {Generalization and Maximum Likelihood from Small Data Sets},
   booktitle = {{IEEE-SP} Workshop on Neural Networks in Signal Processing},
   pages = {(7 pages)},
   year = {1993}
}

Back to Bill Byrne publications.