SpanBERT: Exploring Architecture

Explore the architecture of SpanBERT, focusing on its span-based masking and boundary representation techniques. Understand how SpanBERT combines masked language modeling with span boundary objectives to enhance token prediction accuracy. Learn to apply a fine-tuned SpanBERT model for question-answering using the transformers library pipeline, gaining practical skills for NLP tasks involving text spans.

We'll cover the following...

The span boundary objective
Predicting the masked token
The masked language modeling objective
The loss function of SpanBERT model
Example: Question-answering with SpanBERT
Coding playground

1.Before We Start

2.Starting Off with BERT

3.A Primer on Transformers

Project

4.Understanding the BERT Model

5.Getting Hands-On with BERT

6.Exploring BERT Variants

7.Different BERT Variants

8.BERT Variants—Based on Knowledge Distillation

9.Applications of BERT

10.Exploring BERTSUM for Text Summarization

11.Applying BERT to Other Languages

12.Exploring Sentence and Domain-Specific BERT

13.Working with VideoBERT, BART, and More

14.Conclusion

Project

SpanBERT: Exploring Architecture