Reducing Memory Allocation

Overview

We now know that the tool spends a lot of time with the garbage collector because it’s allocating too much memory. We also know that the bulk of the memory allocation comes from the ReadAll() function of the encoding/csv package, which we’re calling from the csv2float() function.

If we follow the program’s logic, it’s reading all the records from each CSV file into memory and then processing them one at a time with a loop, storing the result in another slice. It’s not doing anything that requires the entire content of the file to be in memory, so we can replace the calls to ReadAll() with the function Read() from the encoding/csv package. This function reads one record at a time, so we can execute it directly inside the loop, preventing the whole file from being read at once.