Learn how to build a machine learning pipeline entirely from scratch, using software engineering best practices.

Building a Machine Learning Pipeline from Scratch.png

docker_setup.tar.gz

omegaconf-code-widget

omegaconf-spa-widget

omegaconf-spa-widget-copy

omegaconf-code-widget-copy

omegaconf_example

dataset_model_examples

logging_example

docstring_example

pipeline_iris_classification

pipeline_autompg_regression

reporting

pipeline_packaging

inference

unit_testing

pipeline_unit_testing

experiment_tracking

argparse_example

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write production-grade training pipelines. This course will provide you with a foundation in ML pipeline development guided by best practices in software engineering.

You’ll start by learning about code organization, style, and conceptual ideas, such as topological sorting of directed acyclic graphs. You’ll dive into the hands-on development of an ML pipeline. You’ll learn some advanced Python concepts and useful libraries. Finally, you’ll learn about testing methodologies and how to package your code into a distributable library.

By the end of this course, you’ll understand what makes a great ML pipeline work, better appreciate software engineering processes, improve your Python knowledge, and learn how to build ML training pipelines. This knowledge will make you more attractive to potential employers and improve your chances of thriving as a new data scientist.

Building a Machine Learning Pipeline from Scratch

Variable names should be meaningful, even if they are a little long.

You should try to reduce the number of lines in a code file, even if that means removing documentation.

Code indentation is important only to the Python interpreter.

It’s okay to use keywords as variable names.

Which statement is true?

There is nothing wrong with the pipeline.

What are some of the issues, if any, in the pipeline shown in this figure?
![Image](/udata/1mgLL0dYl6o/QuizQ2.png)

What are some of the issues, if any, in the pipeline shown in this figure?
<img src="/udata/1mgLL0dYl6o/QuizQ2.png" alt="Image" />

All files, including config files, are in one directory with a flat structure.

Is there anything wrong with the code shown below?
```
mkdir ml_pipeline_tutorial
cd ml_pipeline_tutorial

touch pipeline.py

touch common.yaml
touch datasets.yaml
touch iris_classification.yaml

touch config.py
touch dataset.py
touch dataset_factory.py
touch model.py
touch model_factory.py
touch utils.py

touch iris.py

touch iris_classifier.py

touch csv_mixin.py
touch plotting_mixin.py
touch training_mixin.py
```

Is there anything wrong with the code shown below?
<pre><code>mkdir ml_pipeline_tutorial
cd ml_pipeline_tutorial

touch pipeline.py

touch common.yaml
touch datasets.yaml
touch iris_classification.yaml

touch config.py
touch dataset.py
touch dataset_factory.py
touch model.py
touch model_factory.py
touch utils.py

touch iris.py

touch iris_classifier.py

touch csv_mixin.py
touch plotting_mixin.py
touch training_mixin.py
</code></pre>

`my_function` and `MyNewFunction` don’t follow the same stylistic convention because the former is snake_case, while the latter is CamelCase.

`sum` is Python built-in, so it shouldn’t be used as a variable name.

**(Select all that apply.)** Which are stylistic or lint issues in the code below? 
```
import numpy as np


def my_function(x, y):
    z = x + y
    return x + y


def MyNewFunction(x, y):
    return x - y


if __name__ == "__main__":
    sum = my_function(1, 2)
    diff = MyNewFunction(5, 3)
    print(sum, diff)
```

(Select all that apply.) Which are stylistic or lint issues in the code below?
<pre><code>import numpy as np


def my_function(x, y):
 z = x + y
 return x + y


def MyNewFunction(x, y):
 return x - y


if __name__ == &quot;__main__&quot;:
 sum = my_function(1, 2)
 diff = MyNewFunction(5, 3)
 print(sum, diff)
</code></pre>

Send them the `.py` file and tell them what the dependencies are.

Send them the `.py` file along with a Pip `requirements.txt` or a Conda `environment.yml` that lists the dependencies and their versions.

Imagine that you’ve written the code shown below. A colleague of yours is supposed to test it. How will you share it?
```
import matplotlib.pyplot as plt
import numpy as np

if __name__ == "__main__":
    x = np.arange(0, 10, 1)
    y = np.arange(0, 20, 2)

    plt.figure()
    plt.plot(x, y)
    plt.show()
```

Imagine that you’ve written the code shown below. A colleague of yours is supposed to test it. How will you share it?
<pre><code>import matplotlib.pyplot as plt
import numpy as np

if __name__ == &quot;__main__&quot;:
 x = np.arange(0, 10, 1)
 y = np.arange(0, 20, 2)

 plt.figure()
 plt.plot(x, y)
 plt.show()
</code></pre>

Test your understanding of the lessons in this chapter.

Introduction

Getting Started

Structuring the ML Pipeline

Directed Acyclic Graphs (DAGs)

The ML Library

The Pipeline Core

Extending the Pipeline

Testing

Deployment

Other Considerations

Wrapping Up

Appendix

Final Assessment

Quiz: Structuring the ML Pipeline