Quiz and Summary on Data Ingestion
The chapter delves into data ingestion on AWS, emphasizing architectural choices between batch and streaming methods based on throughput and latency needs. It outlines the use of AWS services like Glue, Kinesis, and DMS for efficient data processing, highlighting strategies for optimizing batch throughput and ensuring pipeline resiliency. The discussion includes orchestration mechanisms, such as Amazon EventBridge, and the benefits of serverless versus provisioned services, ultimately guiding users in selecting appropriate ingestion services for various data sources and workloads.
Summary
This chapter provided a comprehensive examination of data ingestion on AWS, covering architectural decisions, managed and programmable services, orchestration mechanisms, and streaming technologies essential for building production-grade data pipelines.
Batch vs. streaming ingestion fundamentals
The foundational decision between batch and streaming ingestion depends on throughput and latency requirements. Batch ingestion optimizes for high-volume processing in time-bucketed windows, while streaming prioritizes subsecond delivery of individual records. AWS Glue batch jobs scale through DPU allocation for terabyte-scale workloads, whereas Kinesis Data Streams provides real-time ingestion with per-shard throughput limits of 1 MB/s writes and 2 MB/s reads.
Batch throughput depends on a few design choices: use columnar formats such as Parquet to reduce scan costs, choose compression based on the data and query pattern, partition by time when queries filter on time ranges, and target file sizes in the 128–512 MB range to balance parallel reads against the overhead of managing too many small files.