Gibbs sampling belongs to the Markov Chain Monte Carlo (MCMC) methods. These methods are a class of algorithms that are needed for sampling from probability distributions.

Gibbs sampling creates a **Markov chain**, which transitions forward according to some probabilistic rules and then converges. At convergence, **Gibbs sampling** approximates the posterior distribution. Posterior distributions are used for uncertain quantities in the Bayesian Analysis.

Let's say that we have a multivariate probability distribution:

This probability distribution tends to generalize one-dimensional normal distributions to higher dimensions.

The distribution given above has two conditional probabilities, given as follows:

The steps of the algorithm are as follows:

Choose a random value for both variables,

$X$ and$Y$ .Sample from the distribution of

$X$ .Sample a new value for

$Y$ on the$X$ we just computed.Repeat

$1$ and$2$ for$n$ iterations.

We can also do alternative sampling between

Now let's look at some pros and cons of Gibbs sampling.

It is easy to evaluate the conditional distributions.

There is an exact sampling from conjugate conditionals.

There are lower dimensional conditions and it s easy to apply both rejection and importance samplings.

There is always a need to derive conditional probability distributions.

There is also a need to derive random samples from these distributions.

Gibbs sampling may get slow because of no diagonal steps and correlated parameters.

Let's suppose that we have two variables,

X/Y | 0 | 1 |

0 | 0.5 | 0.2 |

1 | 0.3 | 0.3 |

Then,

Similarly, we can write some conditional distributions as follows:

So, we simulate Gibbs sampling if we do as follows:

Take a new value of

$X$ from$Pr(X|Y=y)$ where$y$ is the current value of$Y$ .Take a new value of

$Y$ from$Pr(Y|X=x)$ where$x$ is the current value of$X$ .

Some applications of Gibbs sampling are as follows:

Modeling unobserved data using Data Augmentation

Statistical mechanics

Image processing (Lattice model)

Bayesian statistics

Bioinformatics (analyzing DNA strands)

Immunology

Segregation and survival analysis

Copyright ©2024 Educative, Inc. All rights reserved

TRENDING TOPICS