What is Neural Machine Translation?

Overview

Neural Machine Translation (NMT) uses neural networks to train a framework to translate from one language to another. The semantics of languages differ so starkly that it becomes challenging to prescribe a phrase-to-phrase or word-to-word translation system.

Note: Using neural networks to train a translation system decreases reliance on individual words and instead focuses on the larger context of the sentence

How does NMT work?

Largely, NMT systems tend to contain a series of deep neural networks that can be trained to translate an entire sentence, paragraph or even a document.

Generally, the input sequence is encoded into a sequence of numbers. The resultant sequence of numbers is eventually fed into the decoding network that converts the sequence into the output sequence of words.

This approach is generally known as the Cho encoder-decoder framework.

The input sequence, $X$, and the output sequence, $Y$, can be represented as a sequence of words till $t$ and $n$ respectively.

Consequently, as per the conditional probabilityThe probability that a particular event occurs given some prior knowledge regarding the occurence of another event. and chain rule, the sequence $Y$can be modeled as follows:

NMT systems that comply with the above probabilistic model are known as L2R auto-regressive compliant.

Embedding Layer

Each linguistic token in the input sequence is discretely converted into a vector of its own. $n$discrete tokens are converted into $n$ different vectors of the same length with each vector embedding the original token.

Encoding using Recurrent Neural Networks

The encoder uses an RNN to model the dependent patterns embedded within the input sequence. The corresponding encoding RNN model can be represented as the following:

The resultant encoded sequence, $H$, is then propagated into the decoding network.

Decoding using Recurrent Neural Networks

The decoding framework also uses RNN using the initial state being simply the encoded sequence, $H$, and the decoding history being marked with a sentence-start token, $<start>$. In consecutive iterations, the decoding history is updated to being the previously decoded token and the state is represented as an accumulation of all previously predicted tokens.

The final decoded sequence can be represented as $S$.

Classification Layer

The decoded state, $S$, is mapped into a larger vector, $Z$, encapsulating the entire vocabulary space of the target language. For a vocabulary of size $V$, $Z$would be of dimensions $Z^V$.

Index $i$of vector $Z$would store the relative probability of $i^{th}$word. In order to scale the values down to valid probabilities between 0 and 1, an activation function such as softmax is used.

Advantages over traditional techniques

• More realistic translation: Translation using NMT takes into account the entire context and structure rather than a limited scope of neighboring words and hence is more likely to capture the true essence of the sentence.
• Diminishing need for domain knowledge: For cross-language translation involving slang words and idiomatic expressions, the need for manual rules-based mapping via a human resource is minimized as the model can be trained over such cases using the dataset.
• Cost-effective and Reusable: For languages that have different dialects and regional variants, the rules and mapping need not be defined again. The existing model can be trained using additional data and be tuned to match its respective application.

On the whole, neural machine translation provides a robust framework that forms a one-stop solution for much of the language-translation problems that especially fail to capture the contextual cues. It is widely accepted and used in popular services such as the Google Neural Machine Translation System.