Learn about different cloud environments and tools for building scalable data and model pipelines.

Become a Data Scientist

In this module, we’ll cover different cloud environments and tools for building scalable data and model pipelines. We’ll learn the different data sets and types of models that will be used heavily in everyday production. Throughout the module, we’ll have plenty of exercises and challenges to get us comfortable working with the diverse toolset. Lastly, we’ll explore streaming model workflows that are crucial for building real-time data pipelines.
By the end of this module, we'll have a better understanding of how to build scalable machine learning pipelines in a cloud environment.

Data Science in Production: Building Scalable Model Pipelines

## What is PySpark?
PySpark is a powerful language for both exploratory analysis and building machine learning pipelines. The core data type in PySpark is the Spark dataframe, which is similar to Pandas dataframes but is designed to execute in a distributed environment. 

While the Spark Dataframe API does provide a familiar interface for Python programmers, there are significant differences in the way that commands issued to these objects are executed. 

A key difference is that Spark commands are lazily executed, which means that commands such as `iloc` are not available on these objects. While working with Spark dataframes can seem to constrain us, the benefit is that PySpark can scale to much larger datasets than Pandas. 




# What is PySpark?
PySpark is a powerful language for both exploratory analysis and building machine learning pipelines. The core data type in PySpark is the Spark dataframe, which is similar to Pandas dataframes but is designed to execute in a distributed environment. 

While the Spark Dataframe API does provide a familiar interface for Python programmers, there are significant differences in the way that commands issued to these objects are executed. 

A key difference is that Spark commands are lazily executed, which means that commands such as `iloc` are not available on these objects. While working with Spark dataframes can seem to constrain us, the benefit is that PySpark can scale to much larger datasets than Pandas. 




Introduction to Building Scalable Model Pipelines

Models as Web Endpoints

Models as Serverless Functions

Containers for Reproducible Models

Workflow Tools for Model Pipelines

PySpark for Batch Pipelines

Cloud Dataflow for Batch Modeling

Streaming Model Workflows

Conclusion

A PySpark Primer

What is PySpark?