Learn about different cloud environments and tools for building scalable data and model pipelines.

Become a Data Scientist

In this module, we’ll cover different cloud environments and tools for building scalable data and model pipelines. We’ll learn the different data sets and types of models that will be used heavily in everyday production. Throughout the module, we’ll have plenty of exercises and challenges to get us comfortable working with the diverse toolset. Lastly, we’ll explore streaming model workflows that are crucial for building real-time data pipelines.
By the end of this module, we'll have a better understanding of how to build scalable machine learning pipelines in a cloud environment.

Data Science in Production: Building Scalable Model Pipelines

scikit-learn provides both linear and logistic regression models for making predictions.

scikit-learn provides both linear and logistic regression models for making predictions.

```python 3
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# See the Datasets lesson
bostonDF = ...  

x_train, x_test, y_train, y_test = train_test_split(
  bostonDF.drop(['label'],axis=1),bostonDF['label'],test_size=0.5)

model = LinearRegression()
model.fit(x_train, y_train)
```

The `train_test_split` function will split _____$\%$ of the dataset for testing. 

<pre><code class="language-python">from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# See the Datasets lesson
bostonDF = ... 

x_train, x_test, y_train, y_test = train_test_split(
 bostonDF.drop(['label'],axis=1),bostonDF['label'],test_size=0.5)

model = LinearRegression()
model.fit(x_train, y_train)
</code></pre>
The <code>train_test_split</code> function will split _____<math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">\%</annotation></semantics></math>% of the dataset for testing.

The number of correct predictions over the total number of predictions.

The number of correctly classified outcomes based on different model thresholds.

The number of wrong predictions over the total number of predictions.

The number of wrongly classified outcomes based on different model thresholds.

```python 3 
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import pandas as pd

# Games data set 
gamesDF = pd.read_csv("games-expand.csv")

x_train, x_test, y_train, y_test = train_test_split(
    gamesDF.drop(['label'],axis=1),gamesDF['label'],test_size=0.5)

model = LogisticRegression()
model.fit(x_train, y_train)

print("Accuracy: " + str(model.score(x_test, y_test)))
print("ROC: " + str(roc_auc_score(y_test, 
                    model.predict_proba(x_test)[:, 1] )))
```            
What does the accuracy metric describe in the code above?

<pre><code class="language-python">from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
import pandas as pd

# Games data set 
gamesDF = pd.read_csv(&quot;games-expand.csv&quot;)

x_train, x_test, y_train, y_test = train_test_split(
 gamesDF.drop(['label'],axis=1),gamesDF['label'],test_size=0.5)

model = LogisticRegression()
model.fit(x_train, y_train)

print(&quot;Accuracy: &quot; + str(model.score(x_test, y_test)))
print(&quot;ROC: &quot; + str(roc_auc_score(y_test, 
 model.predict_proba(x_test)[:, 1] )))
</code></pre>
What does the accuracy metric describe in the code above?

The general process for building models with Keras is: 
1. Compile the model 
2. Evaluate the model
3. Fit the model 
4. Set up the structure of the model 

The general process for building models with Keras is:
<ol>
<li>Compile the model</li>
<li>Evaluate the model</li>
<li>Fit the model</li>
<li>Set up the structure of the model</li>
</ol>

Introduction to Building Scalable Model Pipelines

Models as Web Endpoints

Models as Serverless Functions

Containers for Reproducible Models

Workflow Tools for Model Pipelines

PySpark for Batch Pipelines

Cloud Dataflow for Batch Modeling

Streaming Model Workflows

Conclusion

Quiz - Prototype Modeling