HomeCoursesThe Hacker's Guide to Scaling Python

Advanced

8h 30min

Updated 3 weeks ago

The Hacker's Guide to Scaling Python
Save

Learn about Python's scalability, concurrency, and distribution via CPU scaling, event loops, queue-based distribution, and building REST APIs to optimize and deploy high-performance applications.

Join 2.7 million developers at

Overview

Content

Reviews

Python, in all of its greatness, is often dismissed when needing to write performant and distributed applications. It’s considered to be slow and not suited for the task. In this course, you will find that with the right implementation of Python, you can write applications that scale horizontally, perform well, and are distributed. To kick things off, you’ll learn about CPU scaling, concurrency, and event loops, all of which are crucial for implementing a distributed system. After that, you’ll move on to queue-based distribution, lock management, and group memberships, which are fundamental concepts for maintaining availability and consistency in distributed applications. In the last half, you’ll learn how to build a REST API, how to deploy your app to a PaaS, and how to optimize your application's performance. By the end of this course, you’ll be more productive with Python, you’ll be able to write faster applications that are distributed, and you’ll have a great new skill that is highly coveted.

Python, in all of its greatness, is often dismissed when needing to write performant and distributed applications. It’s consider...Show More

Content

99 Lessons1 Project13 Quizzes1 Challenge

Scaling

5 Lessons

Step through scaling Python applications using multithreading, distributing systems, and service-oriented architecture.

Introduction to Scaling

Scaling Across CPUs

Distributed Systems

Service-Oriented Architecture

Quiz on Scaling

CPU Scaling

9 Lessons

Walk through CPU scaling techniques using threads, processes, futures, and optimization tools.

Introduction to CPU Scaling

Using Threads

Using Processes

Using Futures

Advanced Futures Usage

Daemon Processes

Mehdi Abaakouk on CPU Scaling

Quiz on Scaling Across CPUs

Coding Challenge: Scaling Across CPUs

Event Loops

7 Lessons

Work your way through event loops, asynchronous code, and efficient network management using asyncio.

Introduction to Event Loops

Basic Pattern: Blocking Events

Using Asyncio

Server And Client

Naoki Inada on asyncio

Quiz: Event Loops

Coding Challenge: Event Loops

Functional Programming

5 Lessons

Grasp the fundamentals of functional programming in Python, its toolkit, and practical applications.

Introduction to Functional Programming

Functional Programming Toolkit: Maps, Filters, and More

Functional Programming Toolkit: "next", "first", and "itertools"

Quiz: Functional Programming

Coding Challenge: Functional Programming

Queue-Based Distribution

10 Lessons

Dig into Python's queue-based distribution, focusing on RQ and Celery for scalability.

Introduction to Queue-Based Distribution

Celery

Celery: Handling Failures

Celery: Chaining Tasks

Celery: Multiple Queues

Celery: Monitoring

Joshua Harlow on Task Distribution

Quiz: Queue-Based Distribution

Coding Challenge: Queue-Based Distribution

Designing for Failure

5 Lessons

Tackle exception handling, retry strategies, and Tenacity for robust Python applications.

Introduction to Designing for Failure

Naive Retrying

Retrying with Tenacity

Quiz: Designing for Failure

Coding Challenge: Designing for Failure

Create a Web Crawler

Project

Project Walkthrough

1 Lessons

Practice using Celery and Redis to efficiently manage and process tasks.

Project Solution Review

Lock Management

9 Lessons

Step through managing locks to ensure safe access and prevent data corruption.

Introduction to Lock Management

Thread Locks

Processes Locks

Processes Locks: Multiprocessing Locks

Processes Locks: Inter-Processes Locks

Using etcd for Distributed Locking

Using Tooz Locking Abstraction

Quiz: Lock Management

Coding Challenge: Lock Management

Group Membership

9 Lessons

Unpack the core of managing group membership in distributed systems using the Tooz library.

Introduction to Group Membership

Creating, Joining and Leaving Groups

Using Capabilities

Using Watchers Callbacks

Consistent Hash Rings

Partitioner

Alexys Jacob-Monier on Cluster Management

Quiz: Group Membership

Coding Challenge: Group Membership

10.

REST Interfaces

11 Lessons

Master the steps to build scalable and efficient REST APIs with Python frameworks.

Rest Interfaces in Python

The WSGI Protocol

Streaming Data

Using ETag

Asynchronous HTTP API

Fast HTTP Client

Comparison Between Fast HTTP Clients

Testing REST API

Chris Dent on HTTP

Quiz: Building REST API

Coding Challenge: Building REST API

Build a REST API in Flask

Project

11.

Deploying on PaaS

7 Lessons

Break down the steps to deploying Python applications on various PaaS platforms.

Introduction to Deploying on PaaS

Quiz: Deploying on PaaS

12.

Testing Distributed Systems

5 Lessons

Map out the steps for testing, environment setup, and service management in distributed systems.

Introduction to Testing Distributed Systems

Setting Up Environments with tox

Manage External Services with pifpaf

Using Fixtures with pifpaf

Quiz: Testing Distributed Systems

13.

Caching

7 Lessons

Investigate caching to enhance Python app performance, covering local, memoization, and distributed methods.

Introduction to Caching

Local Caching

Memoization

Distributed Caching

Jason Myers on Databases

Quiz: Caching

Coding Challenge: Caching

14.

Performance

8 Lessons

Master the steps to optimize Python performance through profiling, disassembling code, and efficient memory use.

Introduction to performance optimization

Memory and CPU Profiling

Profiling Strategy and a Case

Zero-Copy

Disassembling Code

Victor Stinner on Performance

Quiz: Performance

Coding Challenge: Performance

15.

Conclusion

1 Lessons

Engage in practical projects to apply your scalable Python skills confidently.

Conclusive Remarks

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Course Author:

Julien Danjou

Developed by MAANG Engineers

Every Educative resource is designed by our in-house team of ex-MAANG software engineers and PhD computer science educators — subject matter experts who’ve shipped production code at scale and taught the theory behind it. The goal is to get you hands-on with the skills you need to stay ahead in today's constantly evolving tech landscape. No videos, no fluff — just interactive, project-based learning with personalized feedback that adapts to your goals and experience.

Trusted by 2.7 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI Mock Interviews

Test your skills in a simulated interview setting. Receive personalized feedback based on your performance. Available for Coding Interviews, System Design, and more.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Course

Applied Machine Learning: Deep Learning for Industry

Gain insights into industry-level machine learning by applying advanced TensorFlow and Python techniques. Explore efficient methods for creating scalable models, from data pipelines to deployment and inference.

3 h

advanced

Course

Feature Engineering for Machine Learning

30 m

advanced

Course

Mastering the Art of Programming in Python 3

Learn advanced Python 3 techniques, explore built-in data structures, and delve into functional and object-oriented programming. Discover iterables, generators, and coroutines through projects.

12 h

advanced

Course

Hands-On Generative Adversarial Networks with PyTorch

In this GAN course, learn GAN fundamentals and PyTorch. Explore DCGANs, conditional GANs, image translations, and text-to-image synthesis to master advanced skills for real-world applications.

16 h

advanced

Course

Matplotlib for Python: Visually Represent Data with Plots

Gain insights into visually representing data with Matplotlib in Python. Explore plotting techniques, control axes, create complex layouts, and draw various plots, enhancing your data science skills.

6 h

intermediate

Course

Python for Scientists and Engineers

Gain insights into Python for scientific computing. Explore arrays, plotting, linear equations, and algorithms using NumPy, Matplotlib, SciPy. Delve into applying tools with practical exercises.

13 h

intermediate

Cloud Lab

Speech Synthesis Using Amazon Polly

Learn to use Amazon Polly for text-to-speech synthesis, integrating SSML for natural speech effects and enhancing applications.

1 h

beginner

Course

Applied Machine Learning: Deep Learning for Industry

advanced

3 hour

Course