It isn’t necessary that all of the applications have this WORM requirement. But we are assuming that the blobs that are written can not be modified, instead, we can upload a new version of that blob if needed.

Why blob store?

Blob store is an important component of many data-intensive applications such as YouTube, Netflix, Facebook, etc. The table below displays the blob storage used by some of the well-known applications. These applications generate a huge amount of unstructured data every day. They require a storage solution that is easily scalable, reliable, and highly available where they can store large media files. As the data continuously increase, these applications need to store an unlimited number of blobs.According to some estimates, YouTube requires more than a petabyte of additional storage per day.In a realistic system like Youtube, a video is stored in multiple resolutions. Also, the video in all resolutions is replicated many times across datacenters and regions for availability purposes. That’s why the total storage required per video is not equal to the size of the uploaded video.

Blobs can be as tiny as a few kilobytes for small images to as large as a few terabytes for an archive file of famous movies. Blobs store is optimized for larger files. For small data, we may use key-value store.

According to the AWS FAQ in 2022, S3 allows a single blob of 5 Terabytes.

Requirements

Let’s list down the functional and non-functional requirements of the system.

Functional

Following are the functional requirements of the blob store:

• Create a containerA container is like a folder in a file system used to group blobs. Don’t mix this container with the docker container.: The users should be able to create containers in order to group blobs. For example, if an application wants to store user-specific data, it can store blobs for different user accounts in different containers. Also, a user may want to group video blobs and separate them from a group of image blobs. A single blob store user can create many containers, and each container can have many blobs, as shown in the following illustration. For the sake of simplicity, we assume that we can’t create a container inside a container.

• PUT data: The blob store should allow the users to upload blobs to the created containers.

• GET data: The system should generate a URL for the uploaded blob so that the user can access that blob later through this URL.

• DELETE data: The users should be able to delete a blob. If the user wants to keep the data for a specified period (retention time) of time, it should support it.

• LIST blobs: The user should be able to get a list of blobs inside a specific container.

• DELETE a container: The users should be able to delete a container and all the blobs inside it. ...