Back To Course Home

Getting Started with Google BERT

0% completed

Before we Start

About the course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer

Understanding the Encoder of the Transformer

Self-Attention Mechanism

Understanding the Self-Attention Mechanism

Multi-Head Attention Mechanism

Learning Position with Positional Encoding

Encoder: Feedforward and Add and Norm Component

Putting all the Encoder Components Together

Understanding the Decoder of the Transformer

Masked Multi-Head Attention

Multi-Head Attention

Decoder: Feedforward and Add and Norm Component

Putting All the Decoder Components Together

Putting the Encoder and Decoder Together

Summary: A Primer on Transformers

Exercise: A Primer on Transformers

Exercise: Decoder Architecture

Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-level Byte Pair Encoding & WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model

Extracting Embeddings From All Encoder Layers of BERT

Fine-Tuning BERT for Downstream Tasks

Sentiment Analysis

Natural Language Interface

Question-Answering

Performing Question-Answering with the Fine-Tuned BERT

Named Entity Recognition

Summary: Getting Hands-on with BERT

Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants - Based on Knowledge Distillation Quiz: BERT Variants - Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Applying BERT to Other Languages

Understanding Multilingual BERT Evaluating M-BERT on the NLI task How Multilingual is Multilingual BERT?M-BERT Generalization Effect of Code-Switching and Transliteration The Cross-Lingual Language Model (XLM)The XLM-R Model Language-Specific BERT BETO for Spanish BERTje for Dutch German BERT Chinese BERT Japanese BERT FinBERT for Finnish BERT Models for Italian and Portuguese RuBERT for Russian Summary: Applying BERT to Other Languages Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT Sentence-BERT With a Siamese Network Sentence-BERT With a Triplet Network Use Cases of Sentence-BERT Model Learning Multilingual Embeddings Through Knowledge Distillation Multilingual Sentence-BERT Model Domain-Specific BERT BioBERT Summary: Exploring Sentence and Domain-Specific BERT Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model Pre-training Dataset and Applications of VideoBERT BART Model Noising Techniques Performing Text Summarization with BART Exploring BERT Libraries Building a Document Answering Model Document Summarization Summary: Working with VideoBERT, BART, and More Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Exercise: A Primer on Transformers

Complete this hands-on exercise to test your knowledge of the components of the encoders.

We'll cover the following

Task

Task

Say we have a transformer with 2 encoders. Assuming that we are expanding the components of encoder 2, rearrange the cards to generate the encoder representation of the input sentence:

Get hands-on with 1200+ tech skills courses.