Reduce Memory Usage
Explore techniques to reduce memory usage in Pandas DataFrames by optimizing data types such as converting int64 to int8 and using category types for object columns. Understand how specifying column types during CSV loading can save memory and enhance processing speed with large datasets.
We'll cover the following...
You may have already noticed that when you load a large file into a DataFrame object, it consumes more memory than you thought. This is a weakness of pandas. But, there are still some methods to reduce memory usage. In this lesson, we’ll you some of those methods.
Type conversion
As we mentioned in the lesson Type conversion, int64 takes much more memory than int8. When we load data from a CSV file, if we don’t specify the dtype for each column, the default dtype would be int64, float64, or object. These are all largest data types. If ...