What is Gibbs sampling?

Gibbs sampling belongs to the Markov Chain Monte Carlo (MCMC) methods. These methods are a class of algorithms that are needed for sampling from probability distributions.

Gibbs sampling

Gibbs sampling creates a Markov chain, which transitions forward according to some probabilistic rules and then converges. At convergence, Gibbs sampling approximates the posterior distribution. Posterior distributions are used for uncertain quantities in the Bayesian Analysis.

Algorithm

Let's say that we have a multivariate probability distribution:

This probability distribution tends to generalize one-dimensional normal distributions to higher dimensions.

The distribution given above has two conditional probabilities, given as follows:

The steps of the algorithm are as follows:

1. Choose a random value for both variables, $X$ and $Y$.

2. Sample from the distribution of $X$.

3. Sample a new value for $Y$ on the $X$ we just computed.

4. Repeat $1$ and $2$ for $n$ iterations.

We can also do alternative sampling between $X$ and $Y$.

Pros and cons

Now let's look at some pros and cons of Gibbs sampling.

Pros

1. It is easy to evaluate the conditional distributions.

2. There is an exact sampling from conjugate conditionals.

3. There are lower dimensional conditions and it s easy to apply both rejection and importance samplings.

Cons

1. There is always a need to derive conditional probability distributions.

2. There is also a need to derive random samples from these distributions.

3. Gibbs sampling may get slow because of no diagonal steps and correlated parameters.

Example

Let's suppose that we have two variables, $X$ and $Y$, with a joint distribution as follows:

Then, $p_{X,Y}(0,0)$ can be given as:

Similarly, we can write some conditional distributions as follows:

So, we simulate Gibbs sampling if we do as follows:

1. Take a new value of $X$ from $Pr(X|Y=y)$ where $y$ is the current value of $Y$.

2. Take a new value of $Y$ from $Pr(Y|X=x)$ where $x$ is the current value of $X$.

Applications

Some applications of Gibbs sampling are as follows:

• Modeling unobserved data using Data Augmentation

• Statistical mechanics

• Image processing (Lattice model)

• Bayesian statistics

• Bioinformatics (analyzing DNA strands)

• Immunology

• Segregation and survival analysis