Transformer-based models have gained popularity due to their ability to comprehend and generate natural language. These models manipulate languages by forming high-level linguistic and semantic representations. These representations have been created through unsupervised pretraining on a large text dataset by performing Masked Language Modeling (MLM). In this project, we’ll use the BART model by Facebook to summarize news articles.
In addition to the MLM task, BART is also trained on the Next Sentence Prediction (NSP) task, which it performs with an autoregressive decoder.
We’ll load the model, fine-tune it on a summarization dataset, and finally evaluate it using the ROUGE score. We’ll use the Hugging Face Hub API for access to the models, the PyTorch library for implementing the deep learning logic, the Weights and Biases library to visualize training, and the Evaluate library to evaluate the model.