Search⌘ K
AI Features
Home
Courses
Grokking the AWS Certified Data Engineer – Associate Exam

Performance Tuning and Engine Maintenance

Effective performance tuning and engine maintenance in AWS Glue and Amazon EMR are crucial for ensuring reliable ETL job execution. Key strategies include diagnosing out-of-memory errors through CloudWatch logs, selecting appropriate worker types for compute resources, and optimizing data layout using columnar formats like Parquet. Implementing auto-scaling and managing partition sizes can significantly enhance performance. Additionally, maintaining consistent outcomes requires ongoing monitoring, capacity planning, and the use of job bookmarks for incremental processing. These practices collectively ensure that data pipelines deliver accurate and timely results while minimizing operational issues.

We'll cover the following...

When your CloudWatch alarms fire because an ETL job has failed, the real work begins. For the AWS Certified Data Engineer – Associate exam, knowing how to observe a pipeline is only half the battle. You must also act on what you observe. This lesson puts you in a troubleshooting mindset, where alerts become diagnostic starting points, and tuning decisions determine whether pipelines deliver repeatable business outcomes. The scope covers three pillars that the exam tests directly:

  • Diagnosing failures from Apache Spark logs.

  • Tuning AWS Glue and Amazon EMR compute resources.

  • Applying data layout best practices.

Throughout the lesson, a single running use case ties everything together. An AWS Glue ETL job ingesting order data begins failing with out-of-memory errors visible in Spark driver logs under the /aws-glue/jobs/logs-v2 CloudWatch log group. The logging infrastructure and metric filters you built in the previous lesson now serve as the diagnostic foundation for every tuning action that follows.

Diagnosing Spark out-of-memory errors

AWS Glue ETL jobs execute on Apache Spark, and the single most common performance failure you will encounter, both in production and on the exam, is java.lang.OutOfMemoryError. Understanding where and why this error occurs is the first step toward resolution.

Locating OOM errors in CloudWatch Logs

When a Glue job run fails, Spark writes detailed logs to the CloudWatch log group /aws-glue/jobs/logs-v2. To find the root cause, open CloudWatch Logs Insights and query for java.lang.OutOfMemoryError or the message Container killed by YARN for exceeding memory limits. These two signatures point to fundamentally different problems:

  • Driver OOM: Occurs when too much data is collected to the driver node, typically caused by calling collect() on a large DataFrame or broadcasting an oversized variable. The driver is a single node, so no amount of horizontal scaling fixes this.

  • Executor OOM: Occurs when an individual Spark task processes a partition that exceeds the executor’s available heap memory. This is commonly caused by data skew, where one partition holds a disproportionate share of records, or by an undersized worker type. ...

Attention: The exam frequently presents scenarios where candidates must distinguish between driver OOM and executor OOM. If the question mentions collect() or broadcast joins, the answer involves driver-side fixes. If it mentions skewed partitions or shuffle spills, the answer involves executor memory or repartitioning.