Defining the NMT Model

In this lesson, we’ll define the model from end to end. We are going to implement an encoder-decoder based NMT model equipped with additional techniques to boost performance. Let’s start off by converting our string tokens to IDs.

Converting tokens to IDs

Before we jump to the model, we have one more text processing operation remaining, that is, converting the processed text tokens into numerical IDs. We’re going to use a tf.keras.layers.Layer to do this. Particularly, we’ll be using the StringLookup layer to create a layer in our model that converts each token into a numerical ID. As the first step, let’s load the vocabulary files provided in the data. But before doing so, we’ll define the variable n_vocab to denote the size of the vocabulary for each language: