...

>

System Design: The Distributed Task Scheduler

System Design: The Distributed Task Scheduler

What is a task scheduler?

A task is a unit of computational work requiring resources (CPU, memory, storage, or bandwidth) for a specific duration. For example, uploading media to Facebook or Instagram triggers several background tasks:

  1. Encoding the photo or video into multiple resolutions.

  2. Validating the media for content monetizationContent monetization is a way of leveraging content so that a service can profit from it as users consume it., copyrights, and other policies.

Although these tasks are required to fully process and distribute the content, the upload request returns immediately. Compute-intensive processing runs asynchronously in the background, keeping the user-facing workflow responsive. When a user posts a comment on Facebook, the UI updates optimistically before backend confirmation completes. The distribution of that comment to followers is handled asynchronously by ...