BERT Models for Italian and Portuguese

UmBERTo for Italian

UmBERTo is the pre-trained BERT model for the Italian language by Musixmatch research. The UmBERTo model inherits the RoBERTa model architecture. The RoBERTa is essentially BERT with the following changes in pre-training:

• Dynamic masking is used instead of static masking in the MLM task.

• The NSP task is removed and trained using only the MLM task.

• Training is undertaken with a large batch size.

• Byte-level BPE is used as a ...