Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Data Validation, Deduplication, and Modeling Readiness

Explore how to validate data types, normalize categories, remove duplicates with business rules, and assess target quality to prepare datasets for modeling. This lesson helps you build a defensible, clean dataset that supports reliable machine learning outcomes and meaningful evaluation.

We'll cover the following...

A customer-support ticket table can load into pandas without errors while still being unsafe for modeling. In the ticket records below, resolution_time_minutes mixes numbers with text, priority contains several spellings for the same category, and some rows repeat a ticket. A model would treat those inconsistencies as data rather than mistakes.

Data validation checks whether each column matches its intended meaning, including its type, range, allowed values, and uniqueness. A table that runs is not necessarily valid. df.dtypes reports how pandas stored each column, not whether the stored form makes business sense.

The complete preparation starts by loading the ticket table, inspecting its first rows and types, cleaning each field, removing records under an explicit duplicate rule, and checking whether a target can support a later model.

Python
import pandas as pd
def main() -> None:
    raw_tickets = {
        "ticket_id": [
            "T-101",
            "T-102",
            "T-103",
            "T-104",
            "T-104",
            "T-105",
            "T-106",
        ],
        "created_at": [
            "2026-01-03 09:15:00",
            "2026-01-03 10:05:00",
            "2026-01-04 14:20:00",
            "2026-01-05 08:40:00",
            "2026-01-05 08:40:00",
            "2026-01-06 11:10:00",
            "2026-01-07 16:55:00",
        ],
        "priority": [
            "High",
            " high ",
            "MEDIUM",
            "low",
            "low",
            "Hi",
            "Medium ",
        ],
        "resolution_time_minutes": [
            45,
            "30",
            "unknown",
            120.0,
            120.0,
            15,
            "60",
        ],
        "breached_sla": [
            0,
            0,
            1,
            1,
            1,
            None,
            0,
        ],
    }
    df = pd.DataFrame(raw_tickets)
    print(df.head())
    print(df.dtypes)
if __name__ == "__main__":
    main()

  • Lines 5–51: The table is intentionally “raw”: resolution_time_minutes mixes numbers with strings (including "unknown"), priority includes casing/whitespace variants plus "Hi", and ticket_id repeats "T-104" to create a duplicated row scenario.

  • Line 53: Building the DataFrame from these mixed Python values leads pandas to infer resolution_time_minutes as an object dtype because a single non-numeric value forces the entire column into a generic container type.

  • Lines 55–56: head() and dtypes are printed as the initial validation check, making the mismatch between readable rows and unsafe storage types visible before any cleaning steps.

This code shows that pandas can store resolution_time_minutes as an object column when one value contains text such as unknown. The table is readable, but an average resolution time or a numeric model input would fail or behave incorrectly until we validate and repair the column.

Normalize values and record what changed

Because the raw table exposes mismatched meanings, each conversion needs a check that makes loss visible. pd.to_datetime() parses created_at into dates. pd.to_numeric(errors='coerce') converts valid numeric strings such as 45 into numbers and replaces invalid entries such as unknown with missing values. The missing values are evidence to inspect, not a reason to ...