Introduction to 3D Machine Learning

Overview

In recent years, machine learning and deep learning have taken the world by storm. In particular, advances in artificial neural networks (ANNs) have led to meteoric improvements in computer vision, natural language processing, signal processing, robotics, weather prediction, medical imaging, architecture and construction, design, and more. Name a field, and machine learning is likely to have an impact.

Computer graphics is no different. Another field known for rapid growth and development is the world of computer graphics and 3D art, which have been making even greater strides due to the deep learning revolution. From Deep Learning Super Sampling (DLSS) to Snapchat filters, Apple RoomPlan to Google MediaPipe, and Mesh R-CNN to Neural Radiance Fields (NeRFs), it is truly an exciting time to be in the world of 3D.

In this course, we will learn the basics of computer vision and 3D rendering as it applies to 3D machine learning. This course will introduce:

• Image formation

• Camera models

• Geometry

• Shaders

• Lighting

• Rendering

• 3D data formats

• Common 3D machine learning techniques

We will also introduce the following landmark 3D machine learning models:

• PointNet

• Mesh R-CNN

• Neural Radiance Fields

Now, before we begin, let’s define 3D machine learning.

What is 3D machine learning?

3D machine learning is a field of computational techniques focused on the application of neural networks to 3D data. Some applications include predicting a 3D model from a collection of images, predicting a 3D model from a single image, novel view synthesis from a collection of images, predicting a posed 3D model from an image, point cloud segmentation, and more.

We’ll be using PyTorch3D, a suite of tools for 3D machine learning research. A series of interactive code snippets will be provided to easily follow along with the content of this course.

Who is this course for?

This course is designed for developers interested in solving machine learning problems on 3D data. It provides an overview of the various types of 3D data, the basics of differentiable computer graphics, and a variety of computer vision and machine learning techniques that are commonly applied to 3D data.

To follow along with this course, a basic understanding of software development and machine learning is assumed. Familiarity with rendering and 3D technology is helpful but not required.