Neural Radiance Fields (NeRF)

Overview

Neural Radiance Fields are a massively popular new technique in the field of computer vision. A Neural Radiance Field (NeRF) is a differentiable neural network that models light transport through a continuous 3D scene. NeRFs are perhaps the new state-of-the-art model for novel view synthesis problems. Since their inception in 2020, hundreds of variations on the technique have introduced many new methods and capabilities, such as faster training, relighting, animation, text-to-3D generation, and more.

Introduction to NeRFs

The NeRF architecture attempts to solve the novel view synthesis problem. Given a collection of 2D views of a scene and their 6DoF poses, we wish to render images from unobserved poses. Taking 2D images and poses as input, the NeRF learns an implicit representation of a scene that can be used to predict the appearance of 2D images from previously unseen positions.

How NeRFs work

The NeRF model is a representation of a radiance field, a function that describes the direction of light transport through a continuously-defined 3D space. The 5D radiance field function is parameterized by position $\bold{x}=(x, y, z) \isin \mathbb{R}^3$ and view direction $\bold{d}=(\phi, \theta)$, where $\phi$ is the elevation and $\theta$ is the azimuth of light direction. The view direction $\bf{d}$ is often represented as a 3D unit vector for convenience. Thus, the radiance function maps from a position and view direction to radiance:

where $\bold{c}$ is the color expressed as a vector of $(R, G, B)$ intensities and $\sigma$ is the density at that point. Density $\sigma$ refers to the likelihood of a light ray terminating at that point in space. To use terminology from the study of neural fields, a NeRF is a plenoptic function that maps the position and angle of light passing through a 3D field to color and density.

With a trained NeRF, we can place a virtual camera somewhere we never collected a 2D image and visualize what the scene might look like from that viewpoint. This also enables interesting effects such as smoothly panning a camera through a captured scene. Rendering a scene from a novel view requires the following steps:

1. For each pixel, apply ray marching to gather a finite number of $(\bold{x}, \bold{d})$ ...