Structuring the Pipeline Core
Explore how to build the core of an ML pipeline by organizing tasks with topological sorting of DAGs, using Python function decorators to identify pipeline tasks, and implementing logging and argument parsing to manage experiment tracking effectively.
We'll cover the following...
We'll cover the following...
Now that we have the components required for training a model, it’s time to put them together. We combine these components using what we call the core of the pipeline. The core imports the necessary library modules and calls them in the appropriate order.
The pipeline has several functionalities:
Read command-line user input.
Read user configuration and build a pipeline DAG.
Topologically sort the DAG.
Traverse the DAG and invoke the function corresponding to each vertex.
Log debug and error ...