s3 clone

Monitoring S3

Having visibility of all Amazon S3 resources is important to assess their security posture and take action on potential areas of weakness.

Fetching Metrics

The first step is to fetch the metrics.

Amazon CloudWatch metrics for Amazon S3 can help understand and improve the performance of applications that use Amazon S3. There are several ways that one can use CloudWatch with Amazon S3.

Daily storage metrics for buckets

Monitor bucket storage using CloudWatch, which collects and processes storage data from Amazon S3 into readable, daily metrics. These storage metrics for Amazon S3 are reported once per day and are provided to all customers at no additional cost.

Request metrics

Monitor Amazon S3 requests to quickly identify and act on operational issues. The metrics are available at 1-minute intervals after some latency for processing. These CloudWatch metrics are billed at the same rate as the Amazon CloudWatch custom metrics.

When enabled, request metrics are reported for all object operations. By default, these 1-minute metrics are available at the Amazon S3 bucket level. We can also define a filter for the metrics using a shared prefix, object tag, or access point:

• Access point – Access points are named network endpoints that are attached to buckets and simplify managing data access at scale for shared datasets in S3. With the access point filter, we can gain insights into our access point usage.

• Prefix – Although the Amazon S3 data model is a flat structure, we can use prefixes to infer a hierarchy. A prefix is similar to a directory name that enables us to group similar objects together in a bucket. The S3 console supports prefixes with the concept of folders. If we filter by prefix, objects that have the same prefix are included in the metrics configuration.

• Tags – Tags are key-value name pairs that we can add to objects. Tags help us find and organize objects easily. We can also use tags as a filter for metrics configurations so that only objects with those tags are included in the metrics configuration.

To align these metrics to specific business applications, workflows, or internal organizations, we can filter on a shared prefix, object tag, or access point.

Replication metrics

They monitor the total number of S3 API operations that are pending replication, the total size of objects pending replication, and the maximum replication time to the destination Region. Replication rules that have S3 Replication Time Control (S3 RTC) or S3 replication metrics enabled will publish replication metrics.

Amazon S3 Storage Lens metrics

We can publish S3 Storage Lens usage and activity metrics to Amazon CloudWatch to create a unified view of our operational health in CloudWatch dashboards. S3 Storage Lens metrics are available in the AWS/S3/Storage-Lens namespace. The CloudWatch publishing option is available for S3 Storage Lens dashboards upgraded to advanced metrics and recommendations.

All CloudWatch statistics are retained for a period of 15 months so that one can access historical information and gain a better perspective on how one’s web application or service is performing.