Search⌘ K
AI Features
Home
Courses
Data Science in Production: Building Scalable Model Pipelines

Introduction to Workflow Tools for Model Pipelines

Explore how to manage batch model pipelines using workflow tools such as Airflow. Understand scheduling, dependency handling, and resource provisioning to ensure reliable and automated model workflows in production.

We'll cover the following...

Model pipelines

Model pipelines are usually part of a broader data platform that provides data sources, such as lakes and warehouses, and data stores, such as an application database. When building a pipeline, it’s useful to be able to schedule a task to run, ensure that any dependencies for the pipeline have already been completed, and to backfill historical data if needed. While it’s possible to perform these types of tasks manually, there are a variety of tools that have been developed to improve the management of data science workflows.

...