Search⌘ K
AI Features
Home
Courses
Google FDE Interview Prep

Containing Failure in Multi-Agent Systems

Explore how to bound failure propagation and resource use in multi-agent systems by managing retries, timeouts, and authorization checks. Learn to preserve evidence and human accountability while designing controlled recovery workflows. Practice tracing and defensible terminal states to demonstrate robust failure containment in system design interviews.

We'll cover the following...

A coordinator, specialist client, and tool adapter each permit three total attempts: One failing request can reach the tool 27 times. This lesson bounds failure propagation, resource use, and recovery while preserving evidence and human accountability. In system-design interview practice, demonstrate those controls with a bounded trace and defensible terminal states.

Expand the attempt ledger

Three total attempts means the initial attempt plus two retries. On a persistently failing nested path:

  • Coordinator attempt 1: Three specialist-client attempts, each making three tool-adapter attempts: Nine leaf calls.

  • Coordinator attempt 2: Another nine leaf calls.

  • Coordinator attempt 3: Another nine leaf calls.

The bound is 3 × 3 × 3 = 27 leaf attempts. “Three retries” would mean four total attempts per layer: 4³ = 64.

Actual counts depend on replay scope. Restarting the whole workflow also replays successful inventory work; replayed writes can repeat effects unless their execution contract prevents it.

Three total attempts means the initial attempt plus two retries: nested allowances can produce 3 × 3 × 3 = 27 leaf calls and replay successful inventory work, whereas one retry owner enforces a shared six-call ceiling and the original deadline without requiring all six units to be spent.
Three total attempts means the initial attempt plus two retries: nested allowances can produce 3 × 3 × 3 = 27 leaf calls and replay successful inventory work, whereas one retry owner enforces a shared six-call ceiling and the original deadline without requiring all six units to be spent.

Name one retry owner and inspect client, transport, and adapter defaults. Disable hidden retries or charge them to the shared ledger. Backoff alone leaves multiplication intact.

Preserve E42 readiness

A service-visit planner for equipment E42 delegates reads to existing inventory and safety specialists with distinct source permissions and authority boundaries:

  • Required evidence: A releasable draft needs current inventory evidence and a passing, authoritative safety assessment. Inventory alone cannot establish readiness.

  • Failed invocation: A failed safety call obtained no assessment; it proves neither safety nor absence of an assessment in the source.

  • Dispatch checkpoint: The planner prepares drafts only. A dispatch coordinator reviews a validated draft before separate work-order creation.

  • Safety checkpoint: Missing safety evidence goes to a maintenance safety reviewer, not dispatch approval.

Containment limits resources, result promotion, and authorization through timeouts, retries, loop caps, fallbacks, human checkpoints, and isolation.

Bound the entire request

Set these initial automated limits:

  • Deadline: 1,500 ms from the original start.

  • Remote-call budget: Six admissions shared across specialists, retries, revisions, and remote fallbacks.

  • Read-attempt cap: Two total attempts per specialist.

  • Attempt timeout: At most 400 ms, bounded by remaining request time.

  • Draft-validation cap: Two iterations, each taking 50 ms here.

  • Retry owner: The orchestrator; lower-layer automatic retries are disabled.

A call admission reserves one ...