Reducing Memory Allocation
Explore techniques to optimize memory usage in Go CLI applications by refactoring the CSV reading process. Learn to replace ReadAll with Read to process records one at a time, enable record reuse, and measure performance improvements with benchmarking and profiling tools to achieve faster and more efficient tools.
We'll cover the following...
Overview
We now know that the tool spends a lot of time with the garbage collector because it’s allocating too much memory. We also know that the bulk of the memory allocation comes from the ReadAll() function of the encoding/csv package, which we’re calling from the csv2float() function.
If we follow the program’s logic, it’s reading all the records from each CSV file into memory and then processing them one at a time with a loop, storing the result in another slice. It’s not doing anything that requires the entire content of the file to be in memory, so we can replace the calls to ReadAll()
with the function Read() from the encoding/csv package. This function reads one record at a time, so we can execute it directly inside the loop, preventing the
whole file from being read at ...