Choosing How Data Moves Between Systems
Explore how to choose between files, APIs, change feeds, webhooks, and change data capture when moving data between systems. Learn how snapshots, watermarks, and checkpoints keep data moving without gaps or duplicates. Understand what each interface can promise about freshness and how to recover when a step fails partway.
We'll cover the following...
- Choosing the interface for Morrow’s inventory data
- What an accepted import I17 actually confirms
- Joining snapshot S800 to the change feed without a gap
- Where checkpoint 803 should be saved
- Why webhook H9 cannot replace the poller
- Should forecasting share the inventory queue?
- Morrow’s file-only vs. CDC access
- Practice: Defending Pine Print’s movement and recovery contract
Morrow Supply offers three inventory interfaces: a nightly CSV, a read API with snapshot pagination and an incremental change endpoint, and best-effort webhook hints. Inventory changes must become visible within five minutes during normal operation, and approved database replication access is unavailable. Choosing the movement path means deciding both how quickly changes arrive and who can recover them after a crash.
Choosing the interface for Morrow’s inventory data
Choose API backfill plus incremental polling; optionally use webhooks to discover changes sooner. Nightly-only transfer misses the freshness target, and direct database change data capture (CDC) requires access Morrow has not approved.
Compare patterns against actual source capabilities, freshness, temporal coupling, permissions, recovery requirements, and operating cost:
Files: Scheduled bulk transfer, including snapshots and increments; delivery completion needs an explicit protocol.
Synchronous APIs: Immediate queries or bounded commands. The caller depends on the remote response arriving within its deadline.
Polling: Repeated reads through a supported incremental interface; recovery depends on the continuation contract.
Webhooks: Provider-initiated notifications that reduce discovery delay; completeness depends on delivery and recovery provisions.
Asynchronous messaging: Queues distribute work; retained streams support independent consumption and replay. Messaging decouples sender and receiver availability while adding durable publication, consumption, and recovery responsibilities.
CDC: Database changes captured through approved source or replication access, with snapshot and log-recovery contracts.
Virtualized or federated reads: Queries retrieve ...