Deploying and Hosting Foundation Models
Explore how to deploy and host large foundation models on AWS using options like Amazon Bedrock and SageMaker AI endpoints. Learn to select the best hosting method based on traffic, latency, and data residency needs. Understand key deployment concepts including provisioning, GPU autoscaling, batch inference, and monitoring GPU metrics for scalable, cost-effective production inference.
We'll cover the following...
The Inference Options and Endpoint Requirements lesson matched classic models to SageMaker AI endpoint types. Foundation models (FMs) change the decision. A model with billions of parameters needs GPU memory that a single small instance can’t provide. Pricing can be per token instead of per instance hour. And the same model can often be served three ways: as a managed Amazon Bedrock model, as a model imported into Bedrock, or as open weights on a SageMaker AI endpoint.
The exam tests whether you can read a scenario’s traffic pattern, latency need, data-residency rule, and model origin, and then pick the hosting option that fits. A deployment is done when the model answers through the intended option, capacity matches demand without paying for idle GPUs, and the evidence (job manifests, copy counts, and GPU metrics) confirms both.
The diagram below shows how the main questions narrow the choice.
Amazon Bedrock inference options
Amazon Bedrock hosts the model for you, so the decision is about how capacity is bought. The Amazon Bedrock Foundations and Model Selection lesson introduced on-demand pricing and Provisioned Throughput. This section adds the operational details that separate the four options.
On-demand and cross-Region inference
On-demand inference does not commit. Throughput is limited by per-model account quotas for requests and tokens per minute, and a burst above them returns ThrottlingException.
modelId in InvokeModel or Converse, and Bedrock routes each request to a Region in the profile that has capacity. There are two kinds of profiles:
Geographic profiles (IDs prefixed
us.,eu.,apac., and others) keep processing inside one geography. Choose them when data residency rules apply. ...