Detailed Design of Twitter

Learn the complex storage strategy behind Twitter’s real-time platform. Analyze how specialized systems like Manhattan, Blobstore, and FlockDB are deployed alongside advanced caching and sharded counters to solve challenges like heavy hitters and ensure high availability in a web-scale System Design.

Storage system

Storage is a core component of any real-time system. Twitter employs a polyglot persistence architecture, selecting specific storage modelsA storage model represents a data store’s essential physical aspects. to optimize performance for different services. This section explores how Twitter evolved its storage strategy, moving between various databases and platforms to meet scaling demands.

Note: This lesson draws on insights from Twitter’s technical blogs.

Google Cloud: Twitter uses HDFS (Hadoop Distributed File System) across tens of thousands of servers to host over $300\ PB$ of data. This includes logs (client, Tweet, and timeline events), database backups, and analytics data. Data in HDFS is compressed using LZOLempel–Ziv–Oberhumer (LZO) is a lossless data compression algorithm that is focused on decompression speed. for efficiency. In 2018, Twitter adopted a partly cloudy strategy, migrating data from on-premise Hadoop clusters to Google Cloud. Initially, they moved Ad-hoc clusters and cold storage, while keeping real-time production clusters on-premise. Big data is stored in BigQueryA fully managed and highly scalable serverless data warehouse and accessed via Presto, a distributed SQL query engine.
Manhattan: To handle rapid user growth, Twitter initially attempted to replace MySQL with Cassandra. However, due to specific limitations, they deprecated Cassandra in 2014 and launched Manhattan, a proprietary, real-time, distributed key-value store. Manhattan serves as the backend for tweets, accounts, and direct messages. It runs clusters of varying sizes, handling millions of QPS for heavy read/write traffic. Manhattan uses RocksDB as its storage engine for node-level data retrieval. While it originally handled time-series counters, that functionality has since moved to MetricsDB.

Blob store: Twitter built blob store to store large objects, including photos, videos, and binary files. It also serves as durable storage for in-memory data checkpoints. See the blob store chapter for implementation details.
SQL-based databases: Twitter uses MySQL and PostgreSQL for services requiring strong consistency, such as ad exchanges and campaign management. Vertica is used to query aggregated datasets and power Tableau dashboards. To scale MySQL, Twitter built the Gizzard framework, which handles sharding through partitioning ...