Visualizing Word Embeddings with TensorBoard

When we wanted to visualize word embeddings in the “Word2vec: Learning Word Embeddings” chapter, we manually implemented the visualization with the t-SNE algorithm. However, we also could use TensorBoard to visualize word embeddings. TensorBoard is a visualization tool provided with TensorFlow. We can use TensorBoard to visualize the TensorFlow variables in our program. This allows us to see how different variables behave over time (for example, model loss/accuracy) so we can identify potential issues in our model.

TensorBoard enables us to visualize scalar values (e.g., loss values over training iterations) and vectors (e.g., model’s layer node activations) as histograms. Apart from this, TensorBoard also allows us to visualize word embeddings. Therefore, it takes all the required code implementation away from us if we need to analyze what the embeddings look like. Next, we’ll see how we can use TensorBoard to visualize word embeddings.

Starting TensorBoard

First, we’ll list the steps for starting TensorBoard. TensorBoard acts as a service and runs on a specific port (by default, on 6006). To start TensorBoard, we’ll need to follow these steps:

1. Open up the command prompt (Windows) or terminal (Ubuntu/macOS).

2. Go into the project home directory.

3. If you are using the Python virtualenv, activate the virtual environment where you have installed TensorFlow.

4. Make sure that you can see the TensorFlow library through Python. To do this, follow these steps:

   1. Type in python3; you will get a >>> looking prompt.

   2. Try import tensorflow as tf.

   3. If you can run this successfully, you are fine.

   4. Exit the python prompt (that is, >>> ) by typing exit().

5. Type in tensorboard --logdir=models:

   1. The --logdir option points to the directory where you will create data to visualize

   2. Optionally, you can use --port=<port_you_like> to change the port TensorBoard runs on

6. You should now get the following message: