Mini Map
Search
⌘ K
Log In
System Design Deep Dive: Real-World Distributed Systems*
0%
1.
File Systems
Beyond Local Storage
Custom Card
Visualizing the Bottleneck
The Centralized Master
Hitting the Ceiling
The Distributed Solution
Custom Card
Why Evolve?
DFS Evolution Recap
2.
Google File System (GFS)
Design for Failure
Custom Card
Custom Card
The Ideal Workload
The GFS Trio
The Manager (Master)
The Chunkservers
The Client
Why 64MB Chunks?
Reading: The 2-Step Dance
Custom Card
The Primary Replica
Data Pipelining
Control Flow
Data Flow (The Chain)
Lazy Deletion
Custom Card
Relaxed Consistency
Custom Card
Handling Stale Data
Custom Card
The Master's Memory
Shadow Masters
GFS in a Nutshell
3.
Google Colossus File System
The GFS Bottleneck
Custom Card
Colossus Architecture
The Client Library
The Curators
The Brain: Bigtable
D File Servers
Custom Card
Shared Storage Illusion
Why Bigtable Matters
Custom Card
Metadata Scale Check
Colossus in a Nutshell
4.
Facebook's Tectonic File System
The Specialization Trap
Custom Card
The Brain: ZippyDB
How ZippyDB Scales
Shards & Tiers
Paxos for Consistency
Tectonic Architecture
Metadata Layering
Layer 1: Name Layer
Layer 2: File Layer
Layer 3: Block Layer
Check Your Knowledge
The Multitenancy Challenge
Traffic Groups
Custom Card
Custom Card
Custom Card
Data Integrity
Tectonic Recap
5.
Databases
The Database Dilemma
Google's Evolution
Step 1: Bigtable
Step 2: Megastore
Step 3: Spanner
Custom Card
Knowledge Check
Chapter Recap
6.
Google Bigtable
The Relational Bottleneck
Custom Card
The 4D Data Model
Dimension 1: Row Key
Dimension 2 & 3: Columns
Dimension 4: Time
Knowledge Check
Write Fast: Memtable vs SSTable
The Supporting Cast
Finding Data: The 3-Level Hierarchy
Step 1: Ask Chubby
Step 2: The Root & Meta
Step 3: The User Tablet
Custom Card
Bloom Filters
Custom Card
Summary
7.
Google Megastore
The Storage Dilemma
The Middle Road
High-Level Architecture
Custom Card
Data Model: Root & Child
Scope of Consistency
Synchronous Replication
The Write Lifecycle
Step 1: Leader Selection
Step 2: Consensus
Step 3: Apply & Ack
Custom Card
Key Concepts
8.
Google Spanner
The Database Dilemma
Custom Card
Key Differences
Anatomy of a Spanner Universe
The Zone
Spanserver & Colossus
The Paxos Engine
Data Locality: Interleaving
Understanding Replicas
Custom Card
Time is an Interval
The Sawtooth of Uncertainty
Waiting Out Uncertainty
The CAP Theorem Hack
Custom Card
Snapshot Reads: Time Travel
Distributed Transactions (2PC)
Custom Card
Summary
9.
Key-value Stores
What is a Key-Value Store?
The Hardware Shift
The Single-Core Era
The Multicore Era
The Flash Storage Era
Evolution of Key-Value Systems
Why Rewrite Software?
Recap: KV store
10.
Many-core Key-value Store
Custom Card
Enter the Key-Value Store
Custom Card
The Facebook Scale Challenge
The Bottleneck: Traffic Jams
One Lane, One Lock
Removing the Lock
Rethinking the Hardware
Custom Card
Custom Card
The Verdict: Efficiency Wins
Custom Card
Key Takeaways
11.
Scaling Memcache
The Scaling Challenge
The Onion Architecture
Solving Memory Tetris
The Fragmentation Problem
Enter the Slab Allocator
Adaptive Slabs
Custom Card
The Traffic Jam (Incast)
Thundering Herds
The Stampede
The Lease Ticket
Restoring Order
Custom Card
Custom Card
Warming Up Cold Clusters
The Lag Problem
Remote Markers
Architecture Recap
12.
SILT
The Billion Key Challenge
Custom Card
Custom Card
The SILT Approach: Multi-Store
Stage 1: The Nursery
Stage 2: The Bridge
Stage 3: The Archive
Stage 1: Writing at Speed
Finding Keys in the Log
Shrinking the Index
Quiz: Fast Writes
Stage 2: Freezing Data
Custom Card
Stage 3: The Deep Archive
Tries Without Pointers
Merging for Efficiency
The Lifecycle of a Request
Step 1: Check the Log
Step 2: Check the Middle
Step 3: Check the Archive
System Design Recap
13.
Amazon DynamoDB
The Cloud Database Revolution
Multi-Tenancy: The Apartment Model
Custom Card
DynamoDB Request Flow
Custom Card
Mastering the Primary Key
The Partition Key
The Composite Key
Horizontal Partitioning Basics
Leaders and Replicas
The Replication Group
Consensus (Paxos)
Check Your Knowledge
The Hot Partition Problem
Token Bucket Bursting
Spending Tokens
Tapping the Reserve
Write-Ahead Log Safety
Custom Card
DynamoDB Design Recap
14.
Concurrency Management
Why Concurrency Matters
Custom Card
Visualizing Control Flow
The Conflict
Strategy A: Lock It First
Strategy B: Risk It
Custom Card
Pick the Right Strategy
Recap: Concurrency Management
15.
Two-phase Locking (2PL)
The Concurrency Trap
Reading the Data
The Overwrite Error
What Transactions Do
Custom Card
Two-Phase Locking Basics
Growing Phase Puzzle
What Deadlock Means
Custom Card
Throughput vs Safety
Key Takeaways
16.
Google Chubby Locking Service
The Consensus Challenge
Custom Card
Architecture: The Chubby Cell
How a Write Happens
Step 1: The Request
Step 2: Consensus
Step 3: Success
Custom Card
Custom Card
Sequencers vs Zombie Writers
Read Caching in Chubby
Sessions and KeepAlives
Graceful Handling of Failure
When the Primary Dies
Jeopardy and Grace Period
New Primary, Same Session
Custom Card
Scaling with Chubby Proxies
Summary
17.
ZooKeeper
The Coordination Problem
Custom Card
High-Level Architecture
The Clients
The Ensemble
Znodes Data Model
ZooKeeper Write Path
Write via Follower
Atomic Broadcast (Zab)
Custom Card
Client-Built Primitives
Lock Without Herds
Your Lock Ticket
Who Gets The Lock?
Group Membership
Custom Card
Follower Write Quiz
ZooKeeper Recap
18.
Big Data Processing: Batch to Stream Processing
Data Runs the World
The Evolution of Speed
MapReduce: The Foundation
Spark: The Speed Up
Kafka: Real-Time Flow
Concept: The Working Set
System Match
Recap: Batch to Stream Processing
19.
MapReduce
The Big Data Problem
Custom Card
Enter MapReduce
Custom Card
Visualizing Word Count
Phase 1: Map
Phase 2: Shuffle & Reduce
Under the Hood: Architecture
The Execution Flow
Completing the Job
Custom Card
The Straggler Problem
Optimizing with Combiners
Partitioning Pitfalls
Handling Bad Records
Custom Card
MapReduce Weakness
MapReduce Recap
20.
Spark
The Disk Bottleneck
Iterative Processing
The Old Way: Disk Loop
The Spark Way: RAM Loop
Custom Card
Why Spark is Faster
Data Locality
Speed Check
The Architect: The Driver
The Workhorse: RDDs
Anatomy of an RDD
Partitions
Lineage
Fault Tolerance Strategy
Lazy Evaluation
Transformations vs Actions
Narrow vs Wide
Execution Check
Broadcast Variables
Accumulators
The DAG Scheduler
Managing Memory
Checkpointing
Key Takeaways
21.
Kafka
The Data Stream Problem
Kafka vs. Traditional Messaging
Producer, Broker, Consumer
Ordering in Partitions
How Consumer Groups Scale
Sequential I/O Speed Boost
Zero Copy in Action
The Traditional Copy Path
Kafka’s Faster Copy Path
Why Brokers Stay Stateless
ZooKeeper’s Coordination Role
Delivery Guarantees Quiz
Leader–Follower Replication
Why Kafka Is So Fast
Kafka Deep Dive Recap
22.
Consensus
The Backbone of Distributed Systems
The Two Generals' Problem
The Lost Message
Why Consensus Fails Here
The FLP Impossibility Result
State Machine Replication (SMR)
2PC vs. Paxos/Raft
Recap: Consensus
23.
Understanding Consensus: Two Generals, FLP, & Byzantine Generals
The Consensus Challenge
Custom Card
The Two Generals' Problem
Scenario 1: Attack Message Lost
Scenario 2: Confirmation Lost
Check Your Understanding
The FLP Impossibility
The Impossible Triangle
Living With FLP
The Byzantine Problem
The 3f+1 Rule
Consensus Recap
24.
Two-phase Commit
The Distributed Agreement Problem
Atomicity: All or Nothing
The Squad: Coordinator & Cohorts
How 2PC Works: The Protocol
Phase 1: Prepare / Vote
Phase 2: Commit / Abort
The Point of No Return
Quiz: Who Can Veto?
2PC vs 3PC Trade-offs
Key Takeaways
25.
State Machine Replication
The Single Point of Failure
What is State Machine Replication?
The Failure Spectrum
Anatomy of a State Machine
The Magic of Determinism
Step 1: The Request
Step 2: Processing
Step 3: The Output
Two Pillars of Coordination
How Many Replicas?
Why Ordering Is Hard
Logical Clocks
Replica-Generated IDs
Seen vs. Accepted
Trusting Outputs Safely
Defensive Against Bad Clients
Replacing Faulty Nodes
The Configurator Role
The Catch-Up Problem
Syncing a New Replica
The Log in SMR
Empty Log Slots
Committed Log Entries
Executing the Log
Managing Growing Logs
SMR Recap
26.
Paxos
The Problem of Consensus
The Concept of Slots
An Empty Log
Filling a Slot
Replication
The Three Roles
Ordering with Proposal IDs
The Two-Phase Dance
Phase 1: Prepare
Phase 1: Promise
Phase 2: Accept
Quiz: Higher Promises
Safety Rule: Honor It
Dueling Proposers
Leader to the Rescue
Why Majority Matters
Why Two Phases?
Paxos in a Nutshell
27.
Raft
The Consensus Challenge
Custom Card
Replicated State Machine View
Server Roles in Raft
Follower Role
Candidate Role
Leader Role
Raft Terms as Time
How a Leader Wins
Election Timeout
Requesting Votes
Majority Elects Leader
Election Trigger Quiz
Heartbeats in Raft
Log Replication Steps
Append on Leader
Replicate to Followers
Commit and Apply
Log Matching Property
Log Conflict Quiz
Safe Elections Rule
Safety Over Availability
Config Change Risks
Joint Consensus Idea
Snapshotting the Log
Raft Basics Recap
Home
Courses
System Design Deep Dive: Real-World Distributed Systems*
Consensus (Paxos)
We'll cover the following...
This component is not supported in the current context
...