Speed up Column Operations

In data processing, it is very common to perform overall operations on certain columns. pandas provides lots of methods for us to do this. The following are some methods we could choose. However, the performance of different methods varies greatly. When we have large amounts of data, the gap becomes more pronounced and has a greater impact on our productivity. In this lesson, we want to compare the performance, or running time, of different methods on the same task.

• Iteration by iloc.
• Iteration by .iterrows().
• apply() function.
• Vectorize like Numpy.

The task is very simple: just calculate the sqrt of column a.

Here is the result on my PC, but it varies greatly in different environments. As you can see, Vectorize is the fastest method; however, iterrows() is the slowest method. Vectorize is about 9000 times faster than iterrows(). So, we strongly recommend that you prioritize Vectorize.

Running time of iterrows is 8.340519215s
Running time of iloc is 0.8993627980s
Running time of apply() is 0.032458498000s
Running time of Vectorize is 0.0009303759999s

Here is the code so you can have a try. It may take a few seconds to finish.