Evaluating M-BERT on the NLI task

In the NLI task, the goal of our model is to determine whether a hypothesis is an entailment (true), contradiction (false), or undetermined (neutral) given a premise. Thus, we feed a sentence pair (premise-hypothesis pair) to the model, and it has to classify whether the sentence pair (premise-hypothesis pair) belongs to entailment, contradiction, or is an undetermined class.

Dataset for NLI task

What dataset can we use for this task? For the NLI task, we generally use the Stanford Natural Language Inference (SNLI) dataset. But since we are evaluating M-BERT in this instance, we use a different dataset called cross-lingual NLI (XNLI). The XNLI dataset is based on a MultiNLI dataset. So, first, let's take a look at the MultiNLI dataset.

MNLI dataset

Multi-Genre Natural Language Inference (MultiNLI) is a corpus similar to SNLI. It consists of premise-hypothesis pairs across various genres. A sample of the MultiNLI dataset is shown in the following table: