Twitter has been under the microscope ever since Elon Musk’s acquisition of the company in the fall. Under changing management and with reduced staff, Twitter intends to speed up their timeline service workflow all while cutting costs. With all the focus on the change in leadership and layoffs, I became more interested in examining Twitter from an engineering perspective.
This post will dive into the proposed changes to Twitter’s system design. If you’re curious about how these changes will help engineers reach those goals, keep reading.
I’ll start by covering Twitter’s current architecture in depth. We’ll cover the timeline service, onboarding, people discovery, ad mixing, the distributed key-value store Manhattan, and their tweet ranking service.
Afterwards, we’ll go through the planned changes to the system – what they are and how they’ll help. The biggest and most notable change is a transition to a new timeline mixer called “Home Mixer.” This comes along with updated, more detailed information about other System Design building blocks like:
Then, we’ll take a look at how these changes will affect Twitter and how else Twitter might improve in the future.
At the center of it all is the “Timeline mixer.” This service was developed way back when Twitter first made the switch to algorithmically generated timelines, instead of reverse chronological ones.
Old Twitter used to just send a stream of tweets, retweets and likes with the newest ones at the top. Now, Twitter generates timelines with an algorithmic recommendation service that serves users a totally personalized timeline. A more complicated service, but a better one.
Timeline mixer compiles, or mixes, content from multiple sources to create a holistic timeline. These main content sources are:
But, if you’ve used Twitter you’ll know that relevant, ranked tweets aren’t the only thing that show up on your timeline. The timeline service also mixes in content from several other services. These secondary services are:
Despite the Timeline Mixer being phased out in favor of the new Home Mixer service, the above three smaller services should be largely unchanged. Let’s cover them in more detail.
This service is a unique workflow that introduces new users to the platform. After a user has completed onboarding, nothing from the onboarding service will be used. The core intent for onboarding is to establish users with a base social network. Without anyone to follow or any mutual accounts, Twitter won’t really click for a user.
Facebook has a similar onboarding service. They found that for the majority of users 20 friends was the critical mass. After a user had at least 20 friends, they could return to the platform and have a positive experience. Fewer than 20 friends, and a user would probably bounce after creating their account.
Back in 2010, Twitter unveiled its first iteration of the people discovery service. It was called “Suggestions for you…” and Twitter was ahead of the curve for social networks that aim to grow their users’ connections.
Unlike the onboarding service, people discovery is an ongoing process. After a user reaches the critical mass of connections, a larger network still increases engagement with the platform.
Twitter’s account suggestions are based on algorithms that personalize the service. Here are some connections that they consider:
Twitter’s people suggestion service was one of the first in the field, and when Facebook went after Twitter in the 2016 and 2017, it was one of the services that it aimed to recreate. But that wasn’t the only service that Facebook wanted to recreate. Twitter piloted the approach of treating celebrity accounts or certain celebrity tweets as public. These tweets could be mixed into timelines to increase variety and engagement.
Much like how a Timeline mixer ranks and orders hydrated candidate tweets, an Ad mixer populates a timeline with relevant ads. As Twitter grew and developed more promoted services, their monolithic ad-serving architecture quickly became outdated. We’ll briefly cover Twitter’s current ad funnel and the workflow for ad mixing.
Through trial and error Twitter engineers were able to build a reusable and cohesive ad service capable of all “general purpose tech ad functions.” Twitter identifies these functions as:
Twitter’s ad service is built around these three functions.
You can see the three most common use cases represented at the very bottom of this workflow. This microservice architecture allows for a diversification of concerns, and enables the servers that were once allocated to running the monolithic architecture to be leaner, faster, and more reliable.
Let’s shine a light on a couple other key players in Twitter’s current system.
We’ll start with Manhattan, Twitter’s in-house distributed key-value store. Manhattan is responsible for storing tweets, accounts, direct messages, and more. They developed it after having scalability issues with Cassandra. Cassandra is an open source NoSQL database with many servers that handle large volumes of data.
Since the very nature of Twitter is built around the idea of tweets being served in real-time, a low-latency database is paramount to its success.
Twitter built Manhattan specifically to address requirements like reliability, availability, low-latency, and scalability, but also extensibility, operability, and developer productivity. By providing a large-scale database that is built for the future, Twitter can ensure that developers will be able to store and retrieve whatever data they need without having to worry about it.
Manhattan is separated into four main layers. From top to bottom they are:
Manhattan is configured to allow other 3rd-party storage engines to be plugged in for future use. They’ve taken advantage of this and now implement RocksDB as a storage engine to store and retrieve data in a particular node.
Twitter’s in-house storage engines are:
If you’re interested in learning more about the database, check out Twitter’s engineering blog dedicated to Manhattan.
There, Twitter engineers go much more in-depth about Manhattan, covering why and how they built it.
A recommendation engine is the real backbone of how these mixing services generate timelines. After content is hydrated, a recommendation engine is able to sort each piece of content into a personalized timeline. This engine determines what tweets, accounts, retweets, and Spaces an individual user is likely to find most interesting. By personalizing timelines, Twitter tries to ensure that users have the best possible experience on the platform.
The Timeline mixer features a recommendation engine, but so does the ad mixer. It’s not just relevant Twitter content that needs to be sorted, but ads too. Every part of content fed to users on Twitter is personalized to maximize user interest and engagement.
Much like other platforms (Facebook, YouTube, Netflix, Spotify, etc), bringing the right content to the right users can make or break your service. According to Netflix, 80% of the shows watched come from recommendations. All of these recommendations, for any service, are based on your history with content on the platform.
Before breaking down the architecture of a recommendation engine, let us first address the criteria that they are trained on. There are several key metrics that assess tweets:
Certain actions are weighted differently. Each action is logged and then scored in aggregate to rank the tweet. It gets more complicated when we have to consider how the scoring relates to real users. Since Twitter wants to ensure the best experience possible, tweets should be delivered according to users’ own individual interests, hobbies, and social network and demographic.
It is the job of a recommendation engine to structure and order timelines based on these criteria.
Tweet selection: The first step is to fetch a pool of tweets from the user’s network.
Training data generation: As mentioned above, each tweet will be weighted according to the other users’ engagement. This component is responsible for generating positive and negative examples for training the user engagement prediction models.
Ranker: Receives the pool of tweets and predicts the probability of engagement for each one. This component can be used to rank the total engagement prediction for a tweet or even specific granular categories.
After the ranking process is complete, there is still more to do. In order to give users an appropriate timeline that is representative of their Twitter social network, multiple factors need to be considered.
There are several tweet selection schemes that inform how a timeline is structured.
After tweets are fed to the user through the timeline, the system collects data on what the user engages with to serve as training data to inform future recommendations.
There is much more to say about how Twitter’s recommendation engine works. More areas to explore in detail are the features that a feed-based system should incorporate and the types of logical models that are used to rank candidate tweets. I won’t cover that here, but if you’re interested in diving a little deeper into Twitter’s recommendation engine, the Educative course Grokking the Machine Learning Interview covers the exact aspect of Twitter in depth.
The dotted lines in this diagram represent Twitter’s new proposed read path. The new “Home Mixer” service will be set to replace the Timeline service.
Home Mixer calls on four services:
Now that we’ve had a peek under the hood of Twitter, let’s discuss how these changes will help the platform.
Twitter’s beginning as a reverse chronological “wall” of tweets has now morphed into a much more data-intensive system. As mentioned above, the mixing services generate timelines of tweets, ads, and social recommendations based on multiple ranking heuristics. Despite this, Twitter still needs to serve timelines quickly.
Alongside the typical non-functional requirements of a large social media platform (consistency availability, reliability, and scalability), users expect new tweets to be served to their timelines in near real-time. When a breaking news story is developing, a media event is ongoing, or there’s a new meme, people converge on Twitter. The phrase “live-tweet” describes this platform-unique idea of a real-time stream of consciousness covering a relevant topic.
There are several ways that Twitter streamlines this process. Cached tweets can be quickly retrieved from Memcache, but the timeline system even prioritizes content that is proximally close given a user’s social graph. That said, there is more that Twitter’s new system may be capable of.
The most data-intensive, and budgetarily expensive, service provided by Twitter is building timelines. This cost is the current focus of many Twitter engineers. The old “Timeline mixer” service is being replaced with a significantly faster and simpler “Home Mixer”.
There is very little information publicly available as to how the new system will be faster, but Elon Musk tweeted that the current Timeline mixer is slow because of 1000s of remote procedure calls. In the above diagram of Twitter’s Home Mixer Architecture, you can see that the Timeline Mixer talks to the Timeline Scorer through Thrift remote procedure calls. Thrift is a communication protocol that is known to be fairly slow because of its method of serializing and deserializing data structures.
In the past, I was able to gain 10x or even 20x improvements in speed by removing Thrift and implementing static data structures where memory can be quickly type cast. It’s impossible to say for certain what Twitter’s goals are or how they’ll go about them, but it seems plausible that Twitter’s candidate ranking service, HomeRanker, will be key in achieving a significant increase in speed.
Serving timelines faster allows Twitter to:
I hope you enjoyed this deep dive into Twitter’s changing architecture. Twitter is a complex system and there is a lot more to it than what we went through in this post. If you’re interested in learning more about how to design Twitter from scratch, check out our most advanced System Design course to date: Grokking Modern System Design Interview for Engineers and Managers. It breaks down all of the key building blocks of modern system design, and outlines how to design some of the biggest and most complicated real-world large-scale distributed systems: YouTube, Instagram, Uber, and many more.
As always, happy learning!
Join a community of 1.7 million readers. Enjoy a FREE, weekly newsletter rounding up Educative's most popular learning resources, coding tips, and career advice.