Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Detecting and Treating Outliers

Explore methods to detect outliers in datasets using interquartile range and z-score rules. Learn to interpret flagged values in business context and decide on appropriate treatments. Understand how visualizations like histograms and box plots reveal skewed distributions. This lesson helps you make evidence-based decisions to clean data without losing important signals for analysis.

We'll cover the following...

A purchase amount of 50000 among transactions of 18, 42, and 96 deserves inspection, but it does not deserve automatic deletion. The 50000 charge could contain an extra zero, represent a legitimate VIP purchase, or identify a transaction worth investigating for fraud. An outlier is a value that is unusually extreme relative to the rest of a distribution. Detection rules identify candidates for review. Business context determines what happens next.

The transaction amounts below make the problem visible. Sorting the records and checking the summary statistics shows how a few large amounts can pull the mean upward while leaving most purchases clustered much lower.

Python
import pandas as pd
def main() -> None:
    transactions = pd.DataFrame(
        {
            "transaction_id": [
                1001,
                1002,
                1003,
                1004,
                1005,
                1006,
                1007,
                1008,
                1009,
                1010,
                1011,
                1012,
            ],
            "customer_segment": [
                "consumer",
                "consumer",
                "consumer",
                "consumer",
                "consumer",
                "consumer",
                "consumer",
                "corporate",
                "corporate",
                "corporate",
                "vip",
                "vip",
            ],
            "amount": [
                18,
                42,
                96,
                12,
                27,
                55,
                210,
                800,
                1200,
                12000,
                50000,
                999999,
            ],
        }
    )
    sorted_df = transactions.sort_values("amount", ascending=True).reset_index(drop=True)
    print(sorted_df.to_string(index=False))
    print()
    print(sorted_df["amount"].describe().to_string())
if __name__ == "__main__":
    main()

  • Lines 5–50: The code constructs an in-memory transactions DataFrame with typical small purchases plus extreme amount values (including 50000) so the distribution has a visible right tail to inspect.

  • Line 52: The code sorts by amount and resets the index so the printed table reads as an ordered list of amounts rather than preserving the original row order.

  • Lines 54–56: The code prints the full sorted table followed by amount.describe() (count/mean/std/min/25%/50%/75%/max) to make it easy to see how a few very large values skew the numeric summaries.

This code shows the full range of amount values rather than hiding extreme purchases inside one average. A high amount is unusual compared with the other transactions, but the table alone cannot tell us whether it is invalid. We need a repeatable way to flag unusual records before we inspect their meaning.

Flagging candidates with the IQR rule

Because the sorted amounts show a concentrated middle and a few large values, the interquartile range rule gives us a distribution-based screening method. The interquartile range (IQR) is the distance between the first quartile, Q1, and the third quartile, Q3. Q1 marks the point below which 25% of values fall. Q3 marks the point below which 75% of values fall.

A common rule flags amounts below the lower bound or above the upper bound.

...