Packaging the ML Library
Understand the importance of packaging your ML library to maintain consistent preprocessing and feature engineering across training and inference. Learn to create a Python package with setuptools, build a distributable wheel, and streamline model deployment while reducing errors.
We'll cover the following...
The need for packaging
There are two main parts to any ML project: training and inference. We receive data from some source (historical in the case of training, streaming in the case of inference) and apply our algorithm (training or inference) in both. Then we use the results (evaluation for training, downstream tasks for inference). The diagram below demonstrates the process.
We can see that there are a lot of similarities between the two flows. Preprocessing and feature engineering are exactly the same blocks in ...