Search⌘ K
Home
Courses
Getting Started with Google BERT

BERT Models for Italian and Portuguese

Learn about the architecture and different variants of the UmBERTo and BERTimbau.

We'll cover the following...

UmBERTo for Italian

UmBERTo is the pre-trained BERT model for the Italian language by Musixmatch research. The UmBERTo model inherits the RoBERTa model architecture. The RoBERTa is essentially BERT with the following changes in pre-training:

  • Dynamic masking is used instead of static masking in the MLM task.

  • The NSP task is removed and trained using only the MLM task.

  • Training is undertaken with a large batch size.

  • Byte-level BPE is used as a ...