Diffusion Models

If you are interviewing for a generative AI role, diffusion models are almost guaranteed to come up, since recent breakthroughs such as Stable Diffusion and DALL•E have pushed them to the center of modern generative modeling. Interviewers often ask candidates to explain diffusion models and contrast them with GANs and VAEs to see whether they understand both current techniques and the fundamentals behind older frameworks.

The question is a proxy for assessing how well you grasp the evolution of generative models and the reasons newer approaches have gained traction. In this lesson, we will clarify diffusion models, explain why they have become so relevant, describe how they work, and compare diffusion, GANs, and VAEs.

What is a diffusion model and how does it generate data?

In generative AI, a diffusion model is a type of model that generates data by progressively denoising random noise to create a sample. That’s a mouthful, so let’s unpack it. The term “diffusion” comes from the idea of particles diffusing (spreading out) in a medium—imagine ink spreading in water, eventually turning the water uniformly colored. In the context of AI, we simulate a diffusion-like process on data: we start with a piece of data (say, an image) and gradually add more and more noise until it becomes pure noise. This is called the forward diffusion process, and it’s a bit like destroying the image step by step.

Formally, the noisy sample at the time step $t$ is computed as:

Where:

• $\mathbf{x}_0$: Original clean sample (e.g., an image)

• $\mathbf{x}_t$: Noisy version of the sample at the timestep ...