Design Async Order Processing with Retries, DLQs, and Idempotency
Asynchronous order processing uses a queue, a durable order intent, and idempotent workers so that at-least-once delivery never double-charges a customer. This hands-on lesson adds bounded retries, a dead-letter queue, and failure tests. Covers: Stable orderId and paymentAttemptId, acknowledging after state is recorded, idempotency keys, durable deduplication records, crash reconciliation, DLQ recovery. Practice: Why won't retries double-charge? Prerequisites: Message queues and retries.
In a design interview, a strong answer here is a complete storefront workflow, not a list of AWS services. A checkout spike causes the checkout API to time out after placing orders. Some customers retry, and later duplicate charges appear. We need a design that accepts an order once, processes it asynchronously, and makes repeated delivery safe.
The checkout API first stores an order intent, a durable record that says the customer requested an order and that processing remains incomplete. It then publishes work using a stable orderId. The producer may retry publication after an uncertain network response, but it must reuse that same identifier. A queue accepts the work, a worker processes it, and the worker acknowledges the message only after recording a safe outcome.
Queue delivery and retry ownership
A queue provides decoupling, not exactly-once processing. We therefore assume at-least-once delivery. The queue can redeliver a message when a worker crashes, loses its lease, or does not acknowledge processing. ...