Data Operations and Support II
Understand how to manage data operations and support tasks for AWS data engineering. Learn to analyze logs using CloudWatch and Athena, orchestrate pipelines with Step Functions, monitor workflows for security and reliability, create dashboards in QuickSight, and ensure data consistency with DataBrew. This lesson prepares you to handle practical challenges in operating and supporting scalable AWS data systems.
We'll cover the following...
Question 45
A company uses Amazon CloudWatch Logs to collect application logs from its data processing pipeline. The logs are generated by AWS Lambda functions and AWS Glue jobs. The company wants to analyze the logs to identify patterns in processing errors over the past 30 days. The analysis must support ad hoc queries with SQL syntax and should not require provisioning any infrastructure.
Which solution meets these requirements with the least operational effort?
A. Export CloudWatch Logs to an Amazon S3 bucket using a scheduled export task, then create an AWS Glue Data Catalog table over the exported logs and query them using Amazon Athena.
B. Deploy an Amazon OpenSearch Service cluster, configure a CloudWatch Logs subscription filter to stream logs to OpenSearch, and use OpenSearch Dashboards for log analysis.
C. Use Amazon CloudWatch Logs Insights to run interactive queries directly on the CloudWatch Logs log groups using its built-in SQL-like query syntax.
D. Create CloudWatch Metrics filters to extract error patterns from the logs, then use CloudWatch Metrics to visualize the error trends over the past 30 days.
Question 46
A data engineer is using an Athena notebook with Apache Spark to explore a large ...