Search⌘ K
AI Features
Home
Courses
Grokking the AWS Certified Machine Learning Engineer - Associate

Text Pre-Processing and Embeddings

Explore techniques for preparing free text data through tokenization, normalization, and domain-specific augmentation. Understand how to select and configure embedding models on AWS, ensuring consistency from data indexing to query processing. This lesson guides you in transforming raw text into meaningful vector representations, essential for semantic search, clustering, and various ML applications.

We'll cover the following...

The previous lesson turned numeric and categorical columns into model-ready features with scaling and encoding. Free text and images need a different route. A support ticket, a product description, or a product photo can't be one-hot encoded. Instead, the text is cleaned and tokenized, and an embedding model converts each item into a fixed-length vector whose position captures its meaning. Similar items land close together, which is what semantic search, RAG, clustering, and many classifiers rely on.

Two things decide whether those vectors are useful. Pre-processing must remove noise without removing meaning, and the embedding model must be chosen and configured for the data's language, modality, and target index. A correct pipeline produces vectors of the expected length from the same model and settings for both documents and queries, and nearest-neighbor checks return the items a domain expert would expect.

The diagram below shows where these steps sit between raw data and the vector store.

From raw text and images to vectors
From raw text and images to vectors

Tokenization: How models read text

Models don't read characters or words directly. A tokenizer splits text into tokens, the units a model has a vocabulary for. Classic NLP pipelines use word-level tokens, often with n-grams, as the input to TF-IDF features or to word-vector algorithms such as BlazingText (one of the SageMaker AI built-in algorithms introduced in Chapter 4). Transformer models, including every foundation model and embedding model in Amazon Bedrock, use subword tokenizers: common words stay whole, while rare words, product codes, and drug names are split into several pieces. As a result, domain-heavy text uses more tokens per word than everyday prose.

Token counts matter because every embedding model has an input limit, and what happens at the limit differs by model:

  • Amazon Titan Text Embeddings V2 accepts up to 8,192 tokens or 50,000 characters. AWS gives an average of 4.7 characters per token for English. The response includes inputTextTokenCount, so a pipeline can log the real count.

  • Amazon Titan Multimodal Embeddings G1 accepts only 256 text tokens, which suits product titles and captions, not full documents.

  • Cohere Embed v3 models accept up to 512 tokens per text and at most 2,048 characters per text, with up to 96 texts per call. Text over 2,048 characters must be split before the call. Within that limit, the truncate parameter decides ...