Prep for Quora

• Consistency (eventual consistency - Dr. Qadeer’s lesson?)

---

• Performance: Pycache: in-process caching
• Scalability and performance: Qmessage: handling billions of tasks per day
• Performance: optimizing memcached efficiency
• Performance: asynchronous programming in python: asynq
• Performance: Faster paint times
• Availability: Ensuring quora’s resilience to disaster

Internal details

• Asynq
• MemcacheD latency improvement
• NLP for correcting question
• Fast paint times, client side improvement
• Quora server specs What’s the issue:
• Horizontal scaling is not as simple as adding more master nodes. Communication with other components such as caching can become a problem.
• Network I/O becomes a bottleneck. Even if the number of parallel processes and CPU power is increased, network latency cannot be reduced beyond a certain point.

Add here the server specifications that Quora uses.

Final design

What changed:

• Better machines with more powerful CPU, RAM, and shared cache.
• Include the Master, FastRouter, and Workers within the same machine to get rid of increasing network I/O latency.
• Communication done through UNIX sockets or TCP.
• Same kind of server so horizontal complexity is convenient. However, a good proportion of master and worker processes can be a challenge and depends on the number of features and sever load.

Add here the server specifications that Quora uses.

Components

• Alchemy: a high-performance, scalable, stateless service for feature extraction, generalized for all machine learning applications at Quora. It provides a default feature engineering framework, and is backed by a compute cluster for serving feature extraction requests in real-time. https://quoraengineering.quora.com/Feature-Engineering-at-Quora-with-Alchemy
• HBase (now MyRocks) store all features for ML computations. E.g. answer scores and feature values, ranking answers. https://quoraengineering.quora.com/A-Machine-Learning-Approach-to-Ranking-Answers-on-Quora
• MyRocks is faster and has better p99 latency. HBase was giving an average P99 latency of about 80ms, while MyRocks gave an average P99 latency of under 4ms.
• Asynq: an abstraction service that makes it easier for developers to write batched cache requests (from memcached) instead of writing it manually on a regular basis. https://quoraengineering.quora.com/Asynchronous-Programming-in-Python
• Faster paint times to improve the page loading at client side: https://quoraengineering.quora.com/Faster-Paint-Times
• Periodic jobs running:

Faheem’s comments

How to build its search index

Task scheduler: how to store questions

Schema: how to store and what store to use.

How to store the relationship between questions and answers and how to efficiently query it.: Because you cannot store it in memory by making a linkedlist. Solution has to be searched in the form of a database schema. How will the associated ID be? How to save the answers. Every answer will have an ID and you will you maintain the order of the IDs. How will you update the list if one of the comments is deleted? So how can you colocate information and to what level?

The answer is the partitioning key. We can do range-based sharding because here you can use a common prefix and store everything against that one common prefix. This means we will be able to colocate data because the data’s partition key will be the same. Same partition key means the shard will be the same. Partition key is different from row key.

A key thing to remember is that data is written infrequently and read frequently. So it is suitable to accept a higher writing cost as compared to reading cost. This means the amortized cost will be zero.

key-value store has limitations (check dynamo). We may not be able to store an answer in key-value store and may require a blob for it. We may use a blob anyways for images or videos. If yo are storing in S3, you may want to store the URL to that image in the key-value store. So structured data will tell you about the path of the data.