Statistical Pattern Processing - Module 4F10

The lectures of this part of the course aim to describe the basic concepts of statistical pattern processing and some of the standard techniques used in pattern classification.

Principal lecturers: Professor PC Woodland, Dr. WJ Byrne
Departmental course module page: http://www.eng.cam.ac.uk/teaching/courses/y4/4f10.html
Any queries, problems, or errors in the handouts, please contact me by email: bill.byrne@eng.cam.ac.uk

A related 4th year module is 4F13 Machine Learning

Handouts

• Handout 1: Introduction and Bayes' Decision Theory.
  Statistical pattern processing, Bayesian decision theory, classification cost, ROC curves
  Notes available in [pdf]
• Handout 2: Multivariate Gaussian and Decision Boundaries.
  Multivariate Gaussian PDFs, maximum likelihood estimation, decision boundaries, generalisation
  Notes available in [pdf]
• Handout 3: Gaussian Mixture Models.
  Gaussian mixture models, parameter estimation, introduction to EM.
  Notes available in [pdf]
• Handout 4: Expectation Maximisation.
  Latent variables both continuous and discrete, proof of EM, factor analysis.
  Notes available in [pdf]
• Handout 5: Linear Classifiers.
  Single layer perceptron, perceptron learning algorithm, Fisher's linear discriminant analysis, limitations.
  Notes available in [pdf]
• Handout 6 and 7: Multi-Layer Perceptrons.
  Basic structure, posterior modelling, regression.
  Notes available in [pdf] and [pdf]
• Handout 8: Support Vector Machines.
  Maximum margin classifiers, handling non-separable data, training SVMs, non-linear SVMs, kernel functions.
  Notes available in [pdf]
• Handout 9: Gaussian Processes.
  inear and basis function regression, Gaussian processes, covariance functions, classification
  Notes available in [pdf]
  Carl Rasmussen's notes available in [pdf]
• Handout 10: Relevance Vector Machines and Classification.
  RVM weight prior, training using EM, classification.
  2012 notes available in [pdf]
• Handout 11: Classification and Regression Trees.
  Decision trees, query selection, multivariate decision trees.
  2012 notes available in [pdf] (mortgage examples [ps])
• Handout 12: Non-Parametric Techniques.
  Parzen windows, K-nearest neighbours, nearest neighbour rule.
  2012 notes available in [pdf]
• Handout 13: Speaker Verification and Identification.
  GMMs and MAP estimation, SVM-based verification, dynamic kernels.
  2012 notes available in [pdf]

Examples papers

• The first examples paper for 4F10 statistical pattern processing are available in [pdf].
  Solutions for examples paper 1 are available in [pdf].
• The second examples paper for 4F10 statistical pattern processing are available in [pdf].
  Solutions for examples paper 2 are. available in [pdf].

On-line material

• matlab.zip matlab routines (iris example data).
  • mlpdemo - mlp training on iris data [train: iris_tra, tests: iris_tes]
  • gmmdemo - gmm training on iris data [train: iris_tra, tests: iris_tes]
• A Matlab toolbox for training Gaussian processes
• A Matlab toolbox for training support vector machines
• Gaussian Mixture Model demonstration.
• Speech-Driven Cartoon Animation with Emotions. Example of an SVM application to detection emotion.
• The Lumiere Project. Example of the use of Bayesian Networks for user modelling.
• Improved Detection of Prostate Cancer using CART. Medical example of CART.
• Mortgage Default: Classification Trees Analysis. Financial example of CART.
• Forest Resource Inventory example of K-NN classification.
• Bayesian Network tutrial by Kevin Murphy. This has matlab software available and many other useful references.
• A Tutorial on Support Vector Machines for Pattern Recognition. Knowledge Discovery and Data Mining, by C. J. C. Burges, 1998.
• For more information on Self-Organising Maps research
• For more information on Gaussian Processes research
• Gaussian Processes for Machine Learning by Rasmusen and Williams - software and examples used in lecture notes

Recommended reading

• Judea Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann, San Mateo, CA, 1997. ISBN 1558604790.
• (*) Richard Duda, Peter Hart and David Stork: Pattern Classification, Second Edition, John Wiley & Sons Inc, 2000. ISBN 0471056693
• (*) Christopher Bishop, Neural Networks for Pattern Recognition, Clarendon Press, 1995. ISBN 0198538642
• (*) Christopher Bishop, Pattern Recognition and Machine Learning, Springer 2006.