Transformer Visualization with BertViz

Jesse Vig’s article, "A Multiscale Visualization of Attention in the Transformer ModelThe paper can be accessed at: https://aclanthology.org/P19-3007.pdf" (2019), recognizes the effectiveness of transformer models. However, Jesse Vig explains that deciphering the attention mechanism is challenging. The paper describes the process of BertViz, a visualization tool.

BertViz can visualize attention head activity and interpret a transformer model’s behavior.

BertViz was first designed to visualize BERT and GPT-3 models. In this lesson, we will visualize the activity of a BERT model.

Let’s now run BertViz.

Running BertViz

It only takes five steps to visualize transformer attention heads and interact with them.

Step 1: Importing BertViz and other modules

The notebook BertViz.ipynb imports bertViz, Hugging Face transformers, and the other basic requirements to implement the program:

Get hands-on with 1200+ tech skills courses.