Approximate inference tutorial

This repo covers the main methods for performing approximate inference in Bayesian statistics.

Contents:

1. Markov Chain Monte Carlo (MCMC)
2. Approximate Bayesian Computation (ABC)
3. Variational Inference (VI)
4. Sequential Monte Carlo (SMC)
5. Amortised Inference

Why approximate?

Let's look at Bayes' rule:

expanding the denominator, we have:

where

---

• : The posterior; the probability of the hypothesis (e.g. that a parameter has a certain value) given the data
• : The likelihood of observing/generating the data given the hypothesis
• : The prior probability of the hypothesis
• : The probability of observing the data under all hypotheses

---

It is that makes this equation so difficult to solve in high dimensions (number of parameters to estimate).

We are left in the following situation:

We have a distribution function

where is easy to compute but , the normalising constant, is very hard. Because it is so hard to compute exactly in most complicated (non-conjugate) models we instead approximate it.

Why do we need to compute the evidence?

Because if we want to work out the probability of a particular outcome, we need to normalize by the total probability of the data under all possible hypotheses.

In the finite/discrete case this looks like the familiar formula we learned in school:

In Bayesian inference the same principle applies, but instead of counting outcomes, we sum or integrate over hypotheses, weighting each by its prior plausibility and the likelihood of the data under it:

This term ensures that the posterior is a proper probability distribution (i.e. integrates to 1).

All of the methods covered in this tutorial repo are ways of avoiding having to compute evidence term in its entireity

A sketch to demonstrate how hard computing the evidence can be: