Loading Methods: Scheduled vs. On-Demand
Explore the two main techniques for loading data into repositories within ETL pipelines: scheduled loading and on-demand loading. Understand how each method works, their benefits, and when to apply each based on business requirements to optimize performance and resource allocation.
We'll cover the following...
Besides deciding how to host and deploy the repository, we should also consider how the data will be loaded into it. Or, more specifically, what should trigger data loading into the repository?
There are two main strategies for loading data into a repository:
Scheduled loading
On-demand loading
Scheduled loading
Scheduled loading is a data loading strategy in which data is transferred from its source to the destination repository at a predetermined interval. This interval can be daily, weekly, monthly, or even yearly, depending on the needs of the business. This approach is often used when there’s a need to maintain a consistent and up-to-date view of the data in the data warehouse or data lake. ...