Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Handling Missing Data

Explore strategies for handling missing data within datasets by diagnosing missingness patterns and selecting appropriate treatments like deletion, median imputation, or model-based methods. Understand how to preserve data integrity and the impact of each method on your analysis and modeling outcomes.

We'll cover the following...

A hospital readmission table can contain a recorded readmitted outcome for every patient while lab_value is absent for several patients. That absence is not harmless. Missingness means a field has no recorded value, commonly represented by NaN or NULL.

The first decision is not how to fill the blank. We first ask whether lab_value is absent randomly or whether the absence relates to patient age, clinical process, or readmission. If missing lab results occur more often among patients who are readmitted, removing those rows changes the population used for later analysis.

The code below compares the original patient count, missing-value counts, readmission patterns, and the rows remaining after dropna() removes every patient without a recorded lab result.

Python
import numpy as np
import pandas as pd
def build_patient_df() -> pd.DataFrame:
    df = pd.DataFrame(
        {
            "patient_id": [101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112],
            "age": [67, 72, 58, 81, 49, 76, 63, 55, 70, 60, 85, 52],
            "lab_value": [
                9.1,
                np.nan,
                7.8,
                np.nan,
                8.6,
                10.4,
                np.nan,
                7.2,
                9.7,
                np.nan,
                11.1,
                8.0,
            ],
            "readmitted": [False, True, False, True, False, True, True, False, False, True, True, False],
        }
    )
    return df
def main() -> None:
    df = build_patient_df()
    print("Patient table (one row per patient):")
    print(df)
    print("\nMissing-value counts (per column):")
    print(df.isna().sum())
    print("\nMissing lab_value rate by readmitted outcome:")
    missing_rate_by_outcome = df["lab_value"].isna().groupby(df["readmitted"]).mean()
    print(missing_rate_by_outcome)
    original_row_count = len(df)
    original_readmission_rate = df["readmitted"].mean()
    df_dropped = df.dropna()
    dropped_row_count = len(df_dropped)
    dropped_readmission_rate = df_dropped["readmitted"].mean()
    print("\nRow counts and readmission rates (before vs after dropna()):")
    summary = pd.DataFrame(
        {
            "row_count": [original_row_count, dropped_row_count],
            "readmission_rate": [original_readmission_rate, dropped_readmission_rate],
        },
        index=["original", "after_dropna"],
    )
    print(summary)
if __name__ == "__main__":
    main()

  • Lines 5–28: build_patient_df() builds a small patient table with readmitted recorded for every patient and lab_value missing for some patients.

  • Lines 34–38: The code prints the patient table and the number of missing values in each column.

  • Lines 41–42: df["lab_value"].isna() builds a missingness indicator for the lab, and grouping that boolean by readmitted with .mean() yields the missing-lab rate separately for False and True outcomes.

  • Lines 47–59: dropna() removes every row with any missing value, and the final 2-row summary makes the impact visible by comparing both the ...