Learn about different cloud environments and tools for building scalable data and model pipelines.

Become a Data Scientist

In this module, we’ll cover different cloud environments and tools for building scalable data and model pipelines. We’ll learn the different data sets and types of models that will be used heavily in everyday production. Throughout the module, we’ll have plenty of exercises and challenges to get us comfortable working with the diverse toolset. Lastly, we’ll explore streaming model workflows that are crucial for building real-time data pipelines.
By the end of this module, we'll have a better understanding of how to build scalable machine learning pipelines in a cloud environment.

Data Science in Production: Building Scalable Model Pipelines

To transform our dataframe into vector data types, we can use the ___________ class to combine a range of columns into a single vector column.

To transform our dataframe into vector data types, we can use the ___________ class to combine a range of columns into a single vector column.

When variables in PySpark are transferred from the driver node to workers nodes for distributed operations, a copy of the variable is made.

When variables in PySpark are transferred from the driver node to workers nodes for distributed operations, a copy of the variable is made.

To randomize the value and create more unbalanced bucket sizes

To randomize the value and create more balanced bucket sizes

```python 3

plays_df.createOrReplaceTempView("plays_df")
plays_df = spark.sql("""
  select *, abs(hash(points_id))%1000 as partition_id 
  from plays_df 
""")

```

What is the purpose of the `hash` function on `points_id`?

<pre><code class="language-python">
plays_df.createOrReplaceTempView(&quot;plays_df&quot;)
plays_df = spark.sql(&quot;&quot;&quot;
 select *, abs(hash(points_id))%1000 as partition_id 
 from plays_df 
&quot;&quot;&quot;)

</code></pre>
What is the purpose of the <code>hash</code> function on <code>points_id</code>?

Quiz on machine learning libraries in PySpark. 

Introduction to Building Scalable Model Pipelines

Models as Web Endpoints

Models as Serverless Functions

Containers for Reproducible Models

Workflow Tools for Model Pipelines

PySpark for Batch Pipelines

Cloud Dataflow for Batch Modeling

Streaming Model Workflows

Conclusion

Quiz - Machine Learning in PySpark