Back To Course Home

Getting Started with Google BERT

0% completed

Before We Start

About the course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer

Understanding the Encoder of the Transformer

Self-Attention Mechanism

Understanding the Self-Attention Mechanism

Multi-Head Attention Mechanism

Learning Position with Positional Encoding

Encoder: Feedforward and Add and Norm Component

Putting All the Encoder Components Together

Understanding the Decoder of the Transformer

Masked Multi-Head Attention

Multi-Head Attention

Decoder: Feedforward and Add and Norm Component

Putting All the Decoder Components Together

Putting the Encoder and Decoder Together

Summary: A Primer on Transformers

Exercise: A Primer on Transformers

Exercise: Decoder Architecture

Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to the BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-Level Byte Pair Encoding and WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model Transformers Extracting Embeddings From All Encoder Layers of BERT Fine-Tuning BERT for Downstream Tasks Sentiment Analysis Natural Language Interface Question-Answering Performing Question-Answering with the Fine-Tuned BERT Named Entity Recognition Summary: Getting Hands-On with BERT Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants—Based on Knowledge Distillation Quiz: BERT Variants—Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Applying BERT to Other Languages

Understanding Multilingual BERT Evaluating M-BERT on the NLI task How Multilingual is Multilingual BERT?M-BERT Generalization Effect of Code-Switching and Transliteration The Cross-Lingual Language Model (XLM)The XLM-R Model Language-Specific BERT BETO for Spanish BERTje for Dutch German BERT Chinese BERT Japanese BERT FinBERT for Finnish BERT Models for Italian and Portuguese RuBERT for Russian Summary: Applying BERT to Other Languages Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT Sentence-BERT with a Siamese Network Sentence-BERT with a Triplet Network Use Cases of Sentence-BERT Model Learning Multilingual Embeddings Through Knowledge Distillation Multilingual Sentence-BERT Model Domain-Specific BERT BioBERT Summary: Exploring Sentence and Domain-Specific BERT Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model Pre-training Dataset and Applications of VideoBERT BART Model Noising Techniques Performing Text Summarization with BART Exploring BERT Libraries Building a Document Answering Model Document Summarization Summary: Working with VideoBERT, BART, and More Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Exercise: Decoder Architecture

Hands-on exercise to test your knowledge of the components of the decoder of the transformers.

We'll cover the following

Task

Task

Say we have a transformer with 2 decoders. Assuming that we are expanding the components of decoder 1, place the decoder components in the empty slots on the left so that our transformer generates the decoder representation of the target sentence:

Get hands-on with 1400+ tech skills courses.