Back To Course Home

Getting Started with Google BERT

0% completed

Before we Start

About the course

Starting Off with BERT

Introduction: BERT

A Primer on Transformers

Introduction to the Transformer Understanding the Encoder of the Transformer Self-Attention Mechanism Understanding the Self-Attention Mechanism Multi-Head Attention Mechanism Learning Position with Positional Encoding Encoder: Feedforward and Add and Norm Component Putting all the Encoder Components Together Understanding the Decoder of the Transformer Masked Multi-Head Attention Multi-Head Attention Decoder: Feedforward and Add and Norm Component Putting All the Decoder Components Together Putting the Encoder and Decoder Together Summary: A Primer on Transformers Exercise: A Primer on Transformers Exercise: Decoder Architecture Quiz: A Primer on Transformers

Understanding the BERT Model

Introduction to BERT Model Working of BERT Configurations of BERT Pre-Training the BERT Model The WordPiece Tokenizer Pre-Training Strategies for the BERT Model Masked Language Modeling Next Sentence Prediction Pre-Training Procedure Subword Tokenization Algorithms Byte Pair Encoding Byte-level Byte Pair Encoding & WordPiece Algorithms Summary: Understanding the BERT Model Quiz: Understanding the BERT Model

Getting Hands-On with BERT

Exploring the Pre-Trained BERT Model Transformers Extracting Embeddings From All Encoder Layers of BERT Fine-Tuning BERT for Downstream Tasks Sentiment Analysis Natural Language Interface Question-Answering Performing Question-Answering with the Fine-Tuned BERT Named Entity Recognition Summary: Getting Hands-on with BERT Quiz: Getting Hands-On with BERT

Exploring BERT Variants

Introducion: BERT Variants

Different BERT Variants

ALBERT ALBERT : Training the Model ALBERT : Embeddings Extraction RoBERTa RoBERTa Tokenizer ELECTRA Generator and Discriminator of the ELECTRA Model Training the ELECTRA Model SpanBERT SpanBERT: Exploring Architecture Summary: Different BERT Variants Quiz: Different BERT Variants

BERT Variants—Based on Knowledge Distillation

Knowledge Distillation Training the Student Network DistilBERT Training the Student BERT (DistilBERT)TinyBERT Teacher-Student Architecture Distillation of Transformer Layer Distillation of Embedding and Prediction Layer Distillation Techniques for Pre-training and Fine-tuning Data Augmentation Transferring Knowledge from BERT to Neural Networks The Data Augmentation Methods Summary: BERT Variants - Based on Knowledge Distillation Quiz: BERT Variants - Based on Knowledge Distillation

Applications of BERT

Introduction: Applications of BERT

Exploring BERTSUM for Text Summarization

Text Summarization Fine-Tuning BERT for Extractive Summarization BERTSUM for Extractive Summary BERTSUM for Abstractive Summarization ROUGE Evaluation Metrics Training the BERTSUM Model Summary: Exploring BERTSUM for Text Summarization Quiz: Exploring BERTSUM for Text Summarization

Applying BERT to Other Languages

Understanding Multilingual BERT Evaluating M-BERT on the NLI task How Multilingual is Multilingual BERT?M-BERT Generalization Effect of Code-Switching and Transliteration The Cross-Lingual Language Model (XLM)The XLM-R Model Language-Specific BERT BETO for Spanish BERTje for Dutch German BERT Chinese BERT Japanese BERT FinBERT for Finnish BERT Models for Italian and Portuguese RuBERT for Russian Summary: Applying BERT to Other Languages Quiz: Applying BERT to Other Languages

Exploring Sentence and Domain-Specific BERT

Sentence-BERT Sentence-BERT With a Siamese Network Sentence-BERT With a Triplet Network Use Cases of Sentence-BERT Model Learning Multilingual Embeddings Through Knowledge Distillation Multilingual Sentence-BERT Model Domain-Specific BERT BioBERT Summary: Exploring Sentence and Domain-Specific BERT Quiz: Exploring Sentence and Domain-Specific BERT

Working with VideoBERT, BART, and More

VideoBERT Model

Pre-training Dataset and Applications of VideoBERT

Noising Techniques

Performing Text Summarization with BART

Exploring BERT Libraries

Building a Document Answering Model

Document Summarization

Summary: Working with VideoBERT, BART, and More

Quiz: Working with VideoBERT, BART, and More

Conclusion

What We've Learned

Document Summarization

Learn how to perform document summarization using the ktrain library.

We'll cover the following

Coding playground

Let's explore how to use the ktrain library to perform document summarization. Let's extract content from Wikipedia for a topic of interest and then perform summarization using ktrain.

First, let's install the wikipedia library using pip:

Get hands-on with 1200+ tech skills courses.