Undergraduate Teaching 2022-23

Engineering Tripos Part IIA, 3F8: Inference, 2022-23

Engineering Tripos Part IIA, 3F8: Inference, 2022-23

Not logged in. More information may be available... Login via Raven / direct.

PDF versionPDF version

Leader

Prof Miguel Hernandez-Lobato

Lecturer

Prof Miguel Hernandez-Lobato

Lab Leader

Dr David Krueger

Timing and Structure

Lent Term.

Prerequisites

3F3 Statistical Signal Processing

Aims

The aims of the course are to:

  • Provide a thorough introduction into the topic of statistical inference including maximum-likelihood and Bayesian approaches
  • Introduce inference algorithms for regression, classification, clustering and sequence modelling
  • Introduce basic concepts in optimisation, dynamic programming and Monte Carlo sampling

Objectives

As specific objectives, by the end of the course students should be able to:

  • Understand the use of maximum-likelihood and Bayesian inference and the strengths and weaknesses of both approaches.
  • Implement methods to solve simple regression, classification, dimensionality reduction, clustering and sequence modelling problems.
  • Implement simple optimisation methods (gradient and coordinate descent, stochastic gradient descent) and dynamic programming (Kalman filter or forward algorithm).

Content

Introduction to inference (2L)

  • Revision of maximum likelihood and Bayesian estimation
  • Revision of Bayesian decision theory
  • Outline of the course

Regression (3L)

  • What are regression problems
  • Revision of properties of Gaussian probability density
  • Maximum likelihood and Bayesian fitting of Gaussians
  • Linear regression and non-linear regression

Classification (2L)

  • Classification problems
  • Logistic regression probabilistic model
  • Training logistic regression using optimisation
  • Stochastic optimisation methods
  • Non-linear feature expansions for logistic regression​​

Dimensionality Reduction (2L)

  • What is dimensionality reduction
  • Principal component analysis as minimising reconstruction cost
  • Principal component analysis as inference​

Clustering (3L)

  • What is clustering
  • The k-means algorithm
  • Gaussian Mixture Models
  • The Expectation Maximisation (EM) Algorithm

Sequence models (3L)

  • Sequence modelling problems
  • Markov Models and Hidden Markov models
  • Inference in Hidden Markov Models using dynamic programming

Very Basic Monte Carlo (introduced through the lectures above)

  • Simple Monte Carlo

Further notes

Lecture allocations above are approximate.

Coursework

Title: Logistic Regression for Binary Classification

To implement an algorithm for performing classification, called logistic regression, using gradient descent optimisation.

Learning objectives

  • understand the logistic regression model through visualising predictions 
  • how to apply maximum likelihood and MAP fitting using optimisation
  • how to implement gradient ascent
  • understand how feature expansions can turn linear methods into non-linear methods

Practical information:

  • Sessions will take place in the DPO, during week(s) [TBD].
  • This activity involves a small amount of preliminary work [estimated duration 1hr].

Full Technical Report:

Students will have the option to submit a Full Technical Report.

Booklists

There is no required textbook. However, the material covered is treated excellent recent text books:

Kevin P. Murphy Machine Learning: a Probabilistic Perspective, the MIT Press (2012).

David Barber Bayesian Reasoning and Machine Learning, Cambridge University Press (2012), available freely on the web.

Christopher M. Bishop Pattern Recognition and Machine Learning. Springer (2006)

David J.C. MacKay Information Theory, Inference, and Learning Algorithms, Cambridge University Press (2003), available freely on the web.

Examination Guidelines

Please refer to Form & conduct of the examinations.

 
Last modified: 23/11/2022 08:41