Search⌘ K
AI Features
Home
Courses
Grokking the AWS Certified Machine Learning Engineer - Associate

Validating Data for Generative and Multimodal AI

Explore how to validate generative and multimodal AI datasets with structural and safety checks. Understand using AWS tools like Glue Data Quality, SageMaker Clarify, Amazon Bedrock Guardrails, Comprehend, and Rekognition to ensure data safety, bias mitigation, and class balance for effective model training.

We'll cover the following...

AWS Glue Data Quality rules and SageMaker Clarify bias metrics, covered earlier in this chapter, assume rows and columns. A fine-tuning dataset of prompt–response pairs, a corpus of support transcripts, or a folder of product photos has no column that a rule can check for a missing value or an out-of-range number. The problems are still there: a response that contradicts its source document, a toxic forum post, an image that should never reach a model, or a dataset in which 90% of the examples come from one language or one skin type.

A model trained on that data repeats the problems. A fine-tuned model reproduces unsafe or wrong answers, and a vision model underperforms on the group it rarely saw. Validation is done when every record passes structural and safety checks, flagged records are quarantined with the reason attached, facet and class distributions are measured and within agreed limits, and the measurements are stored as evidence for the training run.

The diagram below shows the validation gate that this lesson builds.

Validation gate for generative and multimodal training data
Validation gate for generative and multimodal training data

Validating prompt–response pairs

The Preparing Data for RAG and FM Customization lesson in Chapter 6 covered the JSON Lines format that fine-tuning expects. A file can have the right format and still teach the model the wrong things. Validate each pair with these structural checks before any safety or quality check:

  • Parse and schema: Every line must be valid JSON with the fields that the customization method expects, such as prompt and completion, or bedrock-conversation-2024 messages that alternate between user and assistant.

  • Empty and truncated records: Remove pairs whose prompt or response is empty, whitespace only, or cut off in the middle of a sentence. Truncated responses often come from exports with a character limit.

  • Length: Check each record against the base model’s token limit, using the same estimate of about 6 characters per token. Records over the limit fail the job or are cut.

  • Duplicates and conflicts: Remove exact duplicates, ...