Preparing Data for RAG and FM Customization
Explore how to prepare and transform data for generative AI workloads including retrieval-augmented generation and foundation model customization. Understand document parsing, chunking strategies, metadata extraction, and the required dataset formats for different foundation model training methods to ensure successful ingestion and model tuning.
The previous lessons prepared tabular features for models that learn from rows and columns. Generative AI workloads consume a different kind of training and grounding data: documents and text. A retrieval-augmented generation (RAG) application needs documents split into well-sized chunks with metadata attached. A customized foundation model (FM) needs JSON Lines records in the exact format that its customization method expects. If the data is prepared badly, the RAG assistant returns half a policy clause or the customization job fails validation before it starts.
This lesson covers both preparation paths.
On the RAG side, you'll learn how to choose a parser for the document types, a chunking strategy for the document structure, and the metadata that lets queries filter and cite results.
On the customization side, you'll learn the dataset that each method needs: labeled prompt–completion pairs for fine-tuning, unlabeled text for continued pre-training, and prompts for distillation.
The preparation is done when the parser matches the document types, the chunks keep complete ideas, each document carries metadata that queries can filter on, and each customization dataset has the format that its job type requires.
The diagram below shows the two paths from raw sources to the services that consume the prepared data.
Parsing documents for a knowledge base
Amazon Bedrock Knowledge Bases (which the RAG and Knowledge Bases lesson in Chapter 8 introduces) ingests documents from Amazon S3 in three steps: parse, chunk, and embed. Parsing turns each file into text, and the choice of parser decides what survives from tables, charts, and scanned pages. A knowledge base data source offers three parsers:
Amazon Bedrock default parser: This parser extracts only text from
.txt,.md,.html,.doc/.docx, .xls/.xlsx, and .pdf files, and it doesn't add usage charges. Figures, charts, and images are lost.Amazon Bedrock Data Automation: This fully managed parser processes multimodal content without prompting. It extracts figures, charts, and tables from PDFs, as well as JPEG and PNG images. It is priced per page or per image.
Foundation model as a parser: A foundation model parses the same multimodal content, and you can customize the parsing prompt, for example, converting every table to Markdown is priced by input and output tokens, and the total size of the parsed files can’t exceed ...