Trusted answers to developer questions
Trusted Answers to Developer Questions

Related Tags

java

What is the HDFS Block?

Fahad Farid

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

The Hadoop Distributed File System (HDFS) stores files in block-sized chunks called data blocks. These blocks are then stored as independent units and are restricted to 128 MB blocks by default. However, they can be adjusted by the user according to their requirements.

Users can adjust block size through the dfs.block.size in the hdfs-site.xml.

If the file size is not a multiple of 128 MB, the last block may be smaller.

Advantages

  1. No limitation on the file size as a file can be larger than any single disk in the network.

  2. Since blocks are of a fixed size, we can easily calculate the number of blocks stored on a given disk. This provides simplicity to the storage subsystem.

  3. Blocks are easy to replicate between DataNodes and, thus, provide fault tolerance and high availability.

  4. Since blocks don’t require storing file metadatasuch as the type of permissions with the blocks, another system can separately handle the metadata.

RELATED TAGS

java

CONTRIBUTOR

Fahad Farid
Copyright ©2022 Educative, Inc. All rights reserved

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Keep Exploring