Handling Failures, Rate Limits, and Silent Errors
Explore how integrations fail, from timeouts and rate limits to calls that succeed without finishing the job. Learn how to decide which failures earn a retry, how idempotency and durable state make retries safe, and how retries share limited capacity. Understand how to detect silent errors and who repairs what cannot be retried.
We'll cover the following...
- What evidence makes S204 ready
- Which failures earn another attempt
- Retries alongside 20 live shipments per minute
- Following a timed-out A1 safely with A2
- How B77 survives a failed dispatch update
- Who repairs S205 and who authorizes replay
- Noticing an HTTP 200 with a pending label
- Practice: Recovering Nacre’s badge job without copying Tern
Tern Dispatch’s workers are healthy, yet shipment S204 is still pending after a courier request timed out. Recovery must establish what happened remotely and finish the shipment without booking it twice. This is the evidence problem behind a reliable integration and behind a convincing interview answer.
S204: Opening Evidence
Observation | What It Establishes | What Remains Unknown |
All workers healthy | Process availability | Business progress |
Courier creation timed out | Caller stopped waiting | Booking effect |
Dispatch screen pending | No visible readiness | Remote label and durable dispatch reference |
All three observations together | Completion is not established | Which business steps remain unfinished |
What evidence makes S204 ready
Suppose Tern’s contract says a shipment is ready only when the courier reports a usable label and dispatch durably stores its reference. Distinguish four acknowledgment boundaries:
Request acceptance: Intake has received the request.
Durable responsibility: The integration has committed the intent and can recover it after a crash; only then does intake acknowledge “accepted for processing.”
Business completion: The courier label is usable, and dispatch durably references it.
User-visible success: The dispatch screen reflects that completed state.
A courier 202 establishes remote acceptance only. A timeout limits the caller’s wait; a remote side effect may already have occurred.
Decide what the available evidence permits the operator to claim.
What does the evidence permit?
Intake has received a shipment request, but the integration has not committed its intent. What may intake acknowledge now?
Say “accepted for processing” if workers are healthy, and commit intent afterward.
Say “accepted for processing,” because receiving the request is enough to take responsibility.
Acknowledge receipt only, and wait for durable intent before saying “accepted for processing.”
An ...