Durability and Correctness in DynamoDB

A large scale database service can run into issues. Faulty code and hardware failures, among other causes, can result in corrupted data. We need to ensure that our system is resilient to these faults. This often requires regular checks and balances that can help us better take care of customers' data. We will aim that data is never lost after it has been committed.

Hardware failures and backups

Modern data centers use commodity serversA commodity computer is easily available and at an attractive price point for an organization. Over the years, the reliability of these nodes has substantially improved. Though, they still fail in one way or the other. We will incorporate measures for situations in which data loss can occur due to hardware failures.

A write-ahead log is very important in our design. It is a log-based (sequential writing in storage with index in memory) data structure that allows us to store large incoming data. The log stored in the storage is essentially the complete data received from incoming requests. The in-memory index helps to access that data in constant time. Even if we lose the in-memory index due to memory failure, we can still use the write-ahead logs to recreate the index.