Asynchronous Event Processing Using API Gateway, SQS, and Fargate

The API Gateway enforces a 29-second integration time-out, which limits how long a backend has to respond before the API returns an error. Short-running requests, whether synchronous or asynchronous, usually complete without issue. However, long-running workloads, such as text analysis, document processing, or data enrichment, can hit the time-out. Even if the backend is asynchronous in nature and continues processing in the background, API Gateway may return an error to the client, making it difficult to reliably handle these tasks. You may request a higher time-out, but there is no guarantee that their processes will always finish within the new limit. To avoid this problem and improve the reliability and responsiveness of applications, these workloads must shift to asynchronous architectures that decouple user requests from backend processing.

In this Cloud Lab, you’ll learn how to build such asynchronous architectures. You’ll start by provisioning a DynamoDB table to store job records and create an SQS queue that provides the fundamental buffering layer between the API Gateway endpoint and the backend processor.

After that, you’ll set up an Amazon ECR repository to store the container image for the processing worker, and then deploy that image using an ECS Fargate service that will continuously poll the queue and handle incoming requests. Once the backend is in place, you’ll create and configure a REST API that accepts new requests asynchronously, immediately returning a job ID rather than waiting for the processing to finish, and later retrieves the final results directly from DynamoDB. After everything is set up, you’ll test the entire architecture using a small frontend application.

After completing this Cloud Lab, you will understand how to design and operate asynchronous processing architectures on AWS that can reliably handle long-running tasks. You will learn how to eliminate API time-outs, enhance the user experience by returning instant responses, and combine services such as API Gateway, SQS, DynamoDB, and Fargate to build scalable and reliable architectures for long-running tasks.

The following is the high-level architecture diagram of the infrastructure you’ll create in this Cloud Lab: