How Much to Buffer?

Explore how to determine the ideal amount of data to buffer before writing to disk in a log-structured file system. Understand the trade-off between disk positioning overhead and transfer rate, and learn how to calculate buffer size to achieve an effective write bandwidth close to the disk's peak capacity.

We'll cover the following...

Now arises the following question: how many updates should LFS buffer before writing to disk? The answer, of course, depends on the disk itself, specifically how high the positioning overhead is in comparison to the transfer rate; see the FFS chapter for a similar analysis.

For example, assume that positioning (i.e., rotation and seek overheads) before each write takes roughly $T_{position}$ seconds. Assume further that the disk transfer rate is $R_{peak}\ MB/s$ . How much should LFS buffer before writing when running on such a disk?

The way to think about this is that every time you write, you pay a fixed overhead of the positioning cost. Thus, how much do you have to write in order to amortize that cost? The more you write, the better (obviously), and the closer you get to achieving peak bandwidth.

To obtain a concrete answer, let’s assume we are writing out $D$ MB. The time to write out this chunk of data ( $T_{write}$ ) is the positioning time $T_{position}$ plus the time to transfer $D$ ( $\frac {D} {R_{peak}}$ ), or:

T_{write} = T_{position} + \frac {D} {R_{peak}}