Search⌘ K
AI Features
Home
Courses
Grokking the AWS Certified Machine Learning Engineer - Associate

Monitoring and Cost Optimization for Generative AI

Explore how to monitor the performance and cost of generative AI models with Amazon Bedrock and SageMaker. Understand key metrics, invocation logs, and evaluation methods to maintain model quality. Learn strategies to attribute and reduce inference expenses, including prompt caching, batch inference, and model selection. Discover techniques to manage embedding computation and vector storage costs for scalable AI applications on AWS.

We'll cover the following...

The earlier lessons in this chapter watched SageMaker AI endpoints: instance metrics, drift and instance-hour bills. A foundation model (FM) called through Amazon Bedrock fails and costs differently. The service is healthy, yet the answers get worse, and the bill grows with every token in every prompt. A prompt that gained 2,000 tokens of retrieved context can double the cost of a request without any error.

This lesson covers the signals that show how FM calls behave (metrics, invocation logs, evaluations), how to attribute and reduce inference cost, and the costs of embeddings and vector storage. The diagram below shows where each signal comes from.

Telemetry and cost signals for a Bedrock application
Telemetry and cost signals for a Bedrock application

Monitoring foundation model calls with CloudWatch

Amazon Bedrock publishes runtime metrics to the AWS/Bedrock namespace for calls through Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream. They need no setup. The following table lists the metrics that matter for operations and cost.

Metric

What It Tells You

Typical Use

Invocations

Requests accepted for processing, including ones that later failed

Traffic baseline

InvocationLatency

Time from request to last token

Latency alarm

TimeToFirstToken

Time to the first token, for streaming calls only

User-perceived speed

InputTokenCount, OutputTokenCount

Tokens in and out

Cost trend and spike alarm

InvocationThrottles

Requests throttled (not counted in Invocations)

Quota or capacity problem

InvocationClientErrors, InvocationServerErrors

Failed invocations

Error-rate alarm

CacheReadInputTokenCount, CacheWriteInputTokenCount

Tokens read from and written to the prompt cache

Cache effectiveness

The ModelId dimension filters by model. A request that fails before Bedrock knows its model is published without that dimension, so a per-model error count can undercount; query without a dimension to see all requests.

CloudWatch generative AI observability builds on these metrics. Its "Model invocations" view is a pre-built dashboard with invocation count, latency, token counts by model, throttles and errors, and it works with any model you call in Bedrock. To see the prompts and responses behind a data point, model invocation logging must be on, as the next section shows. The same feature has an AgentCore view for agents, as the agent monitoring lesson covered. The "Alarm" icon on a widget creates an alarm on that metric.

Token metrics make a cost alarm possible before the invoice arrives. The following command alarms when one model produces more than 5 million output tokens in an hour.

Bash
# AWS CLI
aws cloudwatch put-metric-alarm \
  --alarm-name bedrock-output-token-spike \
  --namespace AWS/Bedrock \
  --metric-name OutputTokenCount \
  --dimensions Name=ModelId,Value=MODEL_ID \
  --statistic Sum --period 3600 --evaluation-periods 1 \
  --threshold 5000000 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions SNS_TOPIC_ARN

Set the threshold from the normal daily pattern, because a fixed number alarms every busy hour. A spike with flat Invocations means longer prompts or answers, not more traffic.

Exam tip: SageMaker metrics such as ModelLatency and GPUUtilization describe instances. Bedrock metrics describe requests and tokens. A question about token usage or throttled FM calls points to AWS/Bedrock.

Invocation logging for audit and attribution

Metrics count tokens but can’t say who sent them or what was asked. Model invocation logging records the full request, the response, and metadata for each call. It is off by default; it applies to one account and ...