Tokenization, Stemming, and Lemmatization
Explore tokenization, stemming, and lemmatization to normalize text for better analysis. Understand how these techniques prepare text data by handling punctuation, casing, stopwords, and word variations to improve topic grouping and sentiment analysis outcomes.
We'll cover the following...
Shipping delayed!!!, Where is my shipment?, and DELIVERED item damaged should be grouped with related customer-support issues, but exact string matching treats each subject as unrelated. Capital letters, punctuation, and different forms of a word prevent a computer from finding useful overlap.
A document is one complete piece of text, such as the ticket subject Refund not received. A token is a word-like unit extracted from a document, such as refund, not, and received. Normalization makes text more consistent before we count it or use it in a model. For these ticket subjects, normalization can turn REFUND?, refund, and Refund!!! into comparable pieces.
The wrong intuition is that text already arrives in a form a model can use. A model cannot directly compare punctuation, capitalization, and word endings as meaning. We first choose how to divide a document into tokens and which differences should matter.
Turning ticket subjects into tokens
Tokenization is the normalization step that converts Item arrived damaged! from one string into a list such as ["item", "arrived", "damaged"]. A simple tokenizer can lowercase text and keep only sequences of letters and digits. That choice removes punctuation noise, but it also makes a decision about what counts as a token.
A library tokenizer may make different choices. For example, a punctuation mark can become its own token, and a contraction such as can't may be divided differently. Neither ...