HomeCoursesData Science in Production: Building Scalable Model Pipelines

4.4

Advanced

Data Science in Production: Building Scalable Model Pipelines

Gain insights into building scalable data and model pipelines, explore different cloud environments, delve into streaming workflows, and discover essential tools for creating real-time data products.

Join 2.7 million developers at

Overview

Content

Reviews

The goal of this course is to provide you with a set of tools that can be used to build predictive model services for product teams. In this course, you’ll start by covering the different cloud environments and tools for building scalable data and model pipelines. You’ll then learn the different data sets and types of models that will be used heavily in everyday production. Throughout the course, you’ll have plenty of exercises and challenges to get you comfortable working with the diverse toolset. Lastly, you’ll explore streaming model workflows which is crucial for building real-time data pipelines that move data between different components in a cloud environment. After working through this course, you will have gained valuable hands-on experience with many of the tools needed to build data products. You will also have a better understanding of how to build scalable machine learning pipelines in a cloud environment.

The goal of this course is to provide you with a set of tools that can be used to build predictive model services for product te...Show More

Content

110 Lessons16 Quizzes4 Cloud Labs

Introduction to Building Scalable Model Pipelines

18 Lessons

Get familiar with building scalable data pipelines, using Python, clouds, coding environments.

Course Overview

Applied Data Science

Python for Scalable Compute

Cloud Environments

Coding Environments

Quiz - Data Science Preliminary Concepts

🛑 Important Note!

Introduction to Datasets

- BigQuery to Pandas

- Kaggle to Pandas

Quiz - Importing Datasets

Prototype Models

- Linear Regression

- Logistic Regression

- Keras Regression

Automated Feature Engineering

Quiz - Prototype Modeling

Conclusion : Data science models, tools and environments

Models as Web Endpoints

12 Lessons

Solve challenges with deploying machine learning models as scalable web endpoints.

Introduction to Models as Web Endpoints

Quiz - Web Services and Persistent Models

Deploying a Web Endpoint

- Gunicorn

- Heroku

Interactive Web Services with Dash

Quiz - Web Deployments

Conclusion : Models as Web Endpoints

Models as Serverless Functions

17 Lessons

Go hands-on with deploying machine learning models using serverless functions on AWS and GCP.

Introduction to Models as Serverless Functions

Managed Services

Cloud Functions (GCP)

- Echo Service

- Cloud Storage (GCS)

Quiz - Managed Services

Lambda Functions (AWS)

- Echo Function

- Simple Storage Service (S3)

- Model Function

- API Gateway

Quiz - Lambda Functions

Conclusion

Create an Echo Function in Lambda

Cloud Lab

Working with S3 in Lambda

Cloud Lab

Working with API in Lambda

Cloud Lab

Containers for Reproducible Models

9 Lessons

Grasp the fundamentals of using containers to ensure reproducible, scalable model deployments.

Introduction to Containers as Reproducible Models

Docker

Orchestration

- AWS Container Registry (ECR)

- AWS Container Service (ECS)

- Load Balancing

Kubernetes on GCP

Quiz - Containers

Conclusion: Containers as Reproducible Models

Working with AWS Container Registry

Cloud Lab

Workflow Tools for Model Pipelines

10 Lessons

Take a closer look at managing and automating model pipelines with workflow tools like Airflow.

Introduction to Workflow Tools for Model Pipelines

Sklearn Workflow

Cron

- Cloud Cron

Quiz - Workflows and Scheduling

Workflow Tools

- Apache Airflow

- Managed Airflow

Quiz - Workflow Tools

Conclusion : Workflow Tools for Model Pipelines

PySpark for Batch Pipelines

25 Lessons

Follow the process of building scalable model pipelines using PySpark in cloud environments.

Introduction to PySpark for Batch Pipelines

Spark Environments

- Spark Clusters

- Databricks Community Edition

Staging Data

Quiz - Spark Environments

A PySpark Primer

- Persisting Dataframes

- Converting Dataframes

Distributed Deep Learning

Distributed Feature Engineering

Quiz - Machine Learning in PySpark

Quiz - GCP Model Pipeline

Conclusion : Working tools for model pipelines

Cloud Dataflow for Batch Modeling

8 Lessons

Master the creation and execution of scalable Cloud Dataflow pipelines for batch modeling.

Introduction to Cloud Dataflow and Batch Modeling

Quiz - Dataflow and Batch Modeling

Conclusion : Cloud Dataflow for Batch Modeling

Streaming Model Workflows

10 Lessons

Learn how to use streaming platforms for scalable real-time machine learning pipelines.

Introduction to Streaming Model Workflows

Spark Streaming

- Apache Kafka

- Sklearn Streaming 1

- Sklearn Streaming 2

Dataflow Streaming

- PubSub

- Natality Streaming

Quiz - Streaming Model Workflows

Conclusion : Streaming model workflows

Course Conclusion

1 Lessons

Look at the evolving journey in data science, emphasizing continuous learning and practice.

Thank You

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Course Author:

Ben Weber

Developed by MAANG Engineers

Every Educative lesson is designed by our in-house team of ex-MAANG software engineers and PhD computer science educators, and developed in consultation with developers and data scientists working at Meta, Google, and more. Our mission is to get you hands-on with the necessary skills to stay ahead in a constantly changing industry. No video, no fluff. Just interactive, project-based learning with personalized feedback that adapts to your goals and experience.

Trusted by 2.7 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Software Developer

Carlos Matias La Borde

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Personalized Interview Prep

Skip the LeetCode grind with a custom roadmap that adapts to your goals. Hands-on practice for Coding Interviews, System Design, and more.

Mock Interviews

Test your skills in a simulated interview setting. Receive personalized feedback based on your performance. Available for Coding Interviews, System Design, and more.

AI Prompt

Build prompt engineering skills. Practice implementing AI-informed solutions.

Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Course

The Hacker's Guide to Scaling Python

Learn about Python's scalability, concurrency, and distribution via CPU scaling, event loops, queue-based distribution, and building REST APIs to optimize and deploy high-performance applications.

8 h 30 m

advanced

Course

Applied Machine Learning: Deep Learning for Industry

Gain insights into industry-level machine learning by applying advanced TensorFlow and Python techniques. Explore efficient methods for creating scalable models, from data pipelines to deployment and inference.

3 h

advanced

Course

Feature Engineering for Machine Learning

30 m

advanced

Course

Hands-On Generative Adversarial Networks with PyTorch

In this GAN course, learn GAN fundamentals and PyTorch. Explore DCGANs, conditional GANs, image translations, and text-to-image synthesis to master advanced skills for real-world applications.

16 h

advanced

Course

Web Security and Access Management: JWT, OAuth2 & OpenId Connect

Gain insights into JWT, OAuth2, and OpenID Connect. Learn about HTTPS, encryption, and handshake techniques, and explore foundational concepts in web API security for enhanced application protection.

5 h

intermediate

Course

Web Application Security for the Everyday Software Engineer

Gain insights into enforcing web app security best practices, such as HTTPS, defending against XSS and clickjacking, managing HTTP cookies, and warding off DDoS attacks.

4 h

intermediate

Course

Building Full-Stack Web Applications with AdonisJs

Gain insights into AdonisJs fundamentals. Delve into routes, controllers, middleware, and hooks. Discover practical examples, illustrations, and quizzes to confidently build your full-stack applications.

intermediate

Course

The Hacker's Guide to Scaling Python

advanced

8 hour 30 min

Course

Applied Machine Learning: Deep Learning for Industry

advanced

3 hour

Course

Feature Engineering for Machine Learning

advanced