FDE System Design: Data Pipelines, Integration and Reliability
Design a reliable data integration pipeline for an FDSE System Design round, where users must know whether the data they act on is fresh and complete. Separate source-owned from locally owned facts, handle duplicate, late and conflicting deliveries with revisions, and commit progress, coverage and state atomically. Tie user-visible freshness to source-confirmed coverage, fence stale workers with leases, map failure signals to owners and actions, and keep access checks current for exports.
We'll cover the following...
- Can dispatchers use a six-minute-old list?
- Which facts may the carrier replace?
- What survives duplicate delivery and interruption?
- What do late events and rejected schemas mean?
- Which signal triggers which action at 10:08?
- What remains true during repair at 10:09?
- Can a revoked dispatcher retrieve an old export?
- Can you specify the inspection feed’s guarantees?
“The worker is alive, but the data is six minutes old. Can the dispatcher trust the exception list?” A fast dashboard response cannot answer that question. A dependable pipeline must connect accepted source changes, durable progress and the freshness shown to the person making a decision. The design below makes that connection inspectable during failure and recovery.
Can dispatchers use a six-minute-old list?
Dispatchers review delayed shipments for their assigned depot and decide whom to contact. The application never automatically reroutes shipments.
The agreed carrier-data freshness limit is five minutes. Beyond that:
Retain the previous list with its original coverage timestamp and a prominent stale state.
Disable carrier-dependent contact recommendations until refresh or an approved manual check.
User-visible freshness must follow committed source coverage, with an explicit consequence when that coverage becomes too old. Neither a live worker nor a fast HTTP response establishes source completeness or current physical shipment truth.
The sources support different claims:
Official role: The FDSE posting describes customer-facing architecture and design, business-critical data, custom applications and end-to-end execution. This supports operational integration’s relevance, not a particular interview format or scoring rubric.
Reported preparation guidance: Third-party guidance emphasizes user-centered pipelines, ingestion, storage, reliability and trade-off defense,not confirmed requirements for an individual candidate.
Practice scope: Public reports do not establish a universal Foundry ...