Search⌘ K
AI Features
Home
Courses
Effective Data Manipulation with pandas

Checking if DataFrames are Equal

Explore methods to verify if two pandas DataFrames are equal, including using equals and pd.testing.assert_frame_equal. Understand common issues such as data type mismatches and rounding errors, and learn practical techniques to diagnose differences for effective data comparison.

We'll cover the following...

The first technique we’ll explore is checking whether two DataFrames are equal. This is especially useful after serializing and deserializing data and, unfortunately, is a little more difficult than it should be. We can use the equals method, which will check if two DataFrames are equal, but if they’re not, diagnosing the problem is hard.

Let’s step through an example with our Dirty Devil data:

Python 3.8
import pandas as pd
url = 'https://github.com/mattharrison/datasets/raw/master'\
      '/data/dirtydevil.txt'
df = pd.read_csv(url, skiprows=lambda num: num <34 or num == 35,
                 sep='\t')
def to_denver_time(df_, time_col, tz_col):
    return (df_
       .assign(**{tz_col: df_[tz_col].replace('MDT', 'MST7MDT')})
       .groupby(tz_col)
       [time_col]
       .transform(lambda s: pd.to_datetime(s)
           .dt.tz_localize(s.name, ambiguous=True)
           .dt.tz_convert('America/Denver'))
    )
def tweak_river(df_):
    return (df_
     .assign(datetime=to_denver_time(df_, 'datetime', 'tz_cd'))
     .rename(columns={'144166_00060': 'cfs',
                      '144167_00065': 'gage_height'})
    )
dd = tweak_river(df)
print(dd)
...