Search⌘ K
Home
Courses
Getting Started with Google BERT

RoBERTa Tokenizer

Explore the RoBERTa tokenizer and its byte-level byte pair encoding method. Learn how it processes spaces using the Ġ character and breaks down words into subwords when necessary. This lesson helps you understand RoBERTa's vocabulary handling and tokenization nuances for improved NLP applications.

We'll cover the following...

Using BBPE as a tokenizer

We know that BERT uses the WordPiece tokenizer. The WordPiece tokenizer works similar to BPE, and it merges the symbol pair based on likelihood instead of frequency. Unlike BERT, RoBERTa uses BBPE as a tokenizer.

The BBPE works very similar to BPE, but instead of using a character-level sequence, it uses a byte-level sequence. We know that BERT uses a vocabulary size of 30,000 tokens, but RoBERTa uses a vocabulary size of 50,000 tokens. Let's explore the RoBERTa tokenizer further.


Import the

...