Operating Agents: Pipelines, Versioning and Monitoring
Learn to manage agent deployments with CI/CD pipelines, version control, vulnerability scans, and automated tests. Monitor key metrics with CloudWatch and X-Ray to detect failures, troubleshoot issues, and track resource consumption.
We'll cover the following...
An agent changes every time someone edits a prompt, swaps a model, or adds a tool, and each change can alter which tools the agent calls and how many tokens it spends. A deployment that overwrites the running agent gives no way back. A deployment with no agent-specific monitoring gives no way to tell why a user got half an answer.
This lesson covers the release path for agents on Amazon Bedrock Agents and Amazon Bedrock AgentCore, then the signals that show how a running agent fails. The diagram below shows the release path and the monitoring loop.
Deployment pipelines for agents
An agent pipeline is a normal CI/CD pipeline with agent-specific tests and a promotion step. The stages are the same on both platforms:
Source and build: A commit to the agent code, prompts, or tool schemas starts the pipeline. For AgentCore Runtime, the build produces an ARM64 container image and pushes it to Amazon ECR.
Scan: Check the image for vulnerabilities (for example with Amazon Inspector) before it can run, as the earlier CI/CD security lesson described.
Create a new version: The pipeline updates the agent and gets an immutable version.
Test the new version: Invoke it with a fixed set of test conversations and assert on the answer and on the tool calls in the trace, as in the prompt testing lesson.
Promote: Point the production alias or endpoint at the tested version.
Observe and roll back: Alarms watch the new version, and a failed check points the alias or endpoint back to the previous version.
AWS CodePipeline, AWS CodeBuild, or a Git-based workflow such as GitHub Actions can run these stages. AWS describes a GitHub Actions pipeline that uses an IAM OpenID Connect identity provider, so no long-lived AWS credentials are stored. Define the agent resources as code with AWS CloudFormation (AWS::BedrockAgentCore::Runtime and AWS::BedrockAgentCore::RuntimeEndpoint for AgentCore) so that environments match and the pipeline’s changes are reviewable.
Pipeline rule: The pipeline promotes a version only after a test passes, and it never edits the version that production serves.