Search⌘ K
AI Features
Home
Courses
Grokking the AWS Certified Data Engineer – Associate Exam

Advanced Exploration and Visualization

Data engineers preparing for the AWS Certified Data Engineer – Associate exam must master advanced data exploration and visualization techniques. This includes using Apache Spark for interactive exploration in Athena notebooks, applying aggregation methods like groupBy and rolling averages, and utilizing AWS Glue DataBrew for data profiling. Visualization is achieved through Amazon QuickSight, which offers dashboarding capabilities. Key distinctions between tools for historical data analysis versus streaming data are emphasized, alongside cost management strategies for Spark and SQL queries. Understanding these concepts is crucial for transforming raw data into actionable insights while ensuring data quality.

We'll cover the following...

Data engineers working with the AWS Certified Data Engineer – Associate (DEA-C01) exam must go beyond writing SQL queries. The exam expects you to know when to shift from structured Athena SQL to interactive, code-driven exploration using Apache Spark, how to apply advanced aggregation techniques that transform raw records into analytical metrics, and which AWS service to choose for visualization. This lesson connects the cost-optimized Athena querying and partitioning strategies you already understand to three new capabilities:

  • Athena notebooks with Apache Spark

  • Aggregation logic, including rolling averages and pivoting

  • Dashboard-driven visualization through AWS Glue DataBrew and Amazon QuickSight

SQL handles well-defined analytical queries efficiently, but Spark notebooks unlock iterative exploration with DataFrames when you do not yet know what you are looking for. Aggregation logic then converts explored data into summary metrics, and visualization tools present those metrics to stakeholders. A recurring exam distractor places streaming tools like Kinesis Data Analytics into scenarios that actually describe aggregation over stored, historical data in S3.

Spark exploration in Athena notebooks

Amazon Athena supports Apache Spark sessions through dedicated Spark-enabled workgroups, allowing data engineers to run interactive PySpark code in notebook cells directly against data stored in Amazon S3. Unlike the Trino-based SQL engine, Athena Spark sessions provision serverless Spark executors behind the scenes, without an EMR cluster to manage.

Configuring a Spark workgroup

Setting up an Athena Spark workgroup involves several parameters that the exam expects you to understand.

  • Executor count (DPUs): Determines the parallel processing capacity of the Spark session, where each DPU provides a fixed amount of vCPU and memory.

  • Session idle timeout: Controls how long an inactive session persists before Athena terminates it, directly affecting cost because billing is based on DPU-hours consumed.

  • IAM role: Grants the Spark session access to S3 buckets containing source data and to the AWS ...