Applying Hugging Face Machine Learning Pipelines in Python.png

dox.tar.gz

NormalCode

JuPyter

JuPy-Translation

JuPy-QA

JuPy-Text-Classification

JuPy-Summarization

JuPy-Text-Generation

JuPy-Similarity

JuPy-Fill-Mask

JuPy-Image-Classification

JuPy-Object-Detection

JuPy-Segmentation

JuPy-Image-to-Text

Hugging Face is a community-driven effort to develop and promote artificial intelligence for a wide array of applications. The organization’s pre-trained, state-of-the-art deep learning models can be deployed to various machine learning tasks.

In this course, you’ll explore the Hugging Face artificial intelligence library with particular attention to natural language processing (NLP) and computer vision. You’ll first explore Hugging Face’s approach to deep learning with specific attention to transformers. You’ll then learn Hugging Face’s pipeline API model and apply various pipelines to unique NLP tasks such as classification, summarization, question answering, and more. You’ll continue with a new set of Hugging Face pipelines for computer vision tasks including object detection and segmentation.

By the end of this course, you’ll be familiar with a wide array of Hugging Face’s pipelines for common machine learning tasks and their implementation in Python using pytorch.

Applying Hugging Face Machine Learning Pipelines in Python

## Overview
Another excellent application of NLP is **Machine Translation**, where a text written in a natural language is automatically translated into another (natural) language. We've all experienced a significant and continuous improvement in the results produced by some common translators like Google and Bing. This improvement can be attributed to transformers, bigger datasets, and better models.


Machine translation has been one of the earliest intended applications of AI.  Despite its initial success in games and some trivial tasks, AI was unable to perform machine translation and was one significant reason behind the first #key# AI winter: Similar to nuclear winter, AI winter refers to the era of reduced funding and interest in AI research. #key#.

The spirit is willing but the flesh is weak

Machine translation drew a lot of attention during the early AI days. There is a folklore story of how the phrase "*The spirit is willing but the flesh is weak*" got translated back (from Russian translation by AI) to "***The vodka is good, but the meat is rotten***." 

Within half a century, today machine translation is able to translate 100+ pairs of languages at human-level performance.


## Translation with Hugging Face

The translation is another sequence-to-sequence task. We perform the translation in Hugging Face as follows:

from transformers import pipeline

en_fr_translator = pipeline("translation_en_to_fr")
translator("It's a pleasant day.")

### Choose a specific model

The example above uses the default translation model, **`T5-base`**. While `T5` is a frequently used model, it is trained in only three languages, and consequently, we need some diverse models.

Hugging Face provides us the luxury of choosing among several translation models as well. As of August 2022, there are 1600+ models on translation alone. These are progressively increasing.

>**Note:** Before using a specific model, we first need to install the text tokenizer/detokenizer #key# **SentencePiece**: SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. #key#.

To use a particular model, we specify it as follows:




from transformers import pipeline

mandarianModel = "Helsinki-NLP/opus-mt-zh-en"       #Most popular translation model on HF
translator = pipeline("translation", model=mandarianModel)
translator("All the variety, all the charm, all the beauty of life is made up of light and shadow.")

### Non-native languages

These classical models are trained in English to some commonly used Indo-European languages. However, what about a language like Lhasa or even an Indo-European language (like Punjabi or Pashto) with many speakers but handicapped by a lack of trained models? 

Fret not! We can use a pre-trained multilingual model in these scenarios and fine-tune it to the desired language.

### Datasets

There are a couple of datasets available if we want to train or fine-tune a model of our own:

- **`opus_books`**: This is a collection of (copyright-free) books translated in sixteen different languages.
- **`code_x_glue_cc_code_to_code_trans`**: It provides some functions in Java and C# as a basic example of translation between programming languages.


## Examples

Let's run some working examples to wrap it up: 

# Overview
Another excellent application of NLP is **Machine Translation**, where a text written in a natural language is automatically translated into another (natural) language. We've all experienced a significant and continuous improvement in the results produced by some common translators like Google and Bing. This improvement can be attributed to transformers, bigger datasets, and better models.

# Translation with Hugging Face

The translation is another sequence-to-sequence task. We perform the translation in Hugging Face as follows:

## Choose a specific model

The example above uses the default translation model, **`T5-base`**. While `T5` is a frequently used model, it is trained in only three languages, and consequently, we need some diverse models.

Hugging Face provides us the luxury of choosing among several translation models as well. As of August 2022, there are 1600+ models on translation alone. These are progressively increasing.

>**Note:** Before using a specific model, we first need to install the text tokenizer/detokenizer #key# **SentencePiece**: SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. #key#.

To use a particular model, we specify it as follows:




## Non-native languages

These classical models are trained in English to some commonly used Indo-European languages. However, what about a language like Lhasa or even an Indo-European language (like Punjabi or Pashto) with many speakers but handicapped by a lack of trained models? 

Fret not! We can use a pre-trained multilingual model in these scenarios and fine-tune it to the desired language.

## Datasets

There are a couple of datasets available if we want to train or fine-tune a model of our own:

- **`opus_books`**: This is a collection of (copyright-free) books translated in sixteen different languages.
- **`code_x_glue_cc_code_to_code_trans`**: It provides some functions in Java and C# as a basic example of translation between programming languages.


# Examples

Let's run some working examples to wrap it up: 

Get an overview of machine translation and learn to perform it using Hugging Face.

Introduction

NLP

Computer Vision

Conclusion

Appendix

Machine Translation

Overview

Translation with Hugging Face

Choose a specific model

Non-native languages

Datasets

Examples