Data Science and Machine Learning Interview Handbook/

...

/

Data Cleaning Techniques

Data Cleaning Techniques

Cover important techniques for data cleaning and automation that data analysis interviewees should prepare for.

We'll cover the following...

No analysis or model can succeed without clean data. In this lesson, we’ll learn how to structure your thinking and solutions for common interview questions around cleaning, transforming, and validating data in both real-time and offline workflows. Let’s get started.

Handling messy data

You’re given a raw dataset filled with inconsistencies—missing values, outliers, structural errors, and duplications. An interviewer asks how you’d clean and prepare this data for analysis while preserving data quality.

What is your process for dealing with messy data?

This question is frequently asked at LinkedIn, Spotify, and Stripe for analytics or data science roles especially any company working with user-generated data (e.g., edtech platforms, surveys, and IoT).

Sample answer

Let’s explore a sample answer that touches upon the key dimensions of “messy” data and emphasizes a multi-step process. Ideally, your approach also highlights attributes such as data integrity and quality, and mentions specific techniques that you use for the types of data you are experienced in (tabular, image, etc.).

  1. Missing data:

    1. Start by identifying any missing values within the dataset using techniques like NA Detection.

    2. Then apply data imputation techniques such as mean/median/mode imputation, or k-nearest neighbors imputation to fill in the missing values with plausible estimates.

  2. Duplicate data:

    1. Use duplicate detection algorithms to identify any duplicate records.

    2. Then typically apply fuzzy matching or record linkage to see how successful they are at eliminating these duplicates, ensuring data integrity, or moving to other techniques if these aren’t able to accurately catch all the duplicates.

  3. Data from different sources:

    1. Perform data integration using ETL (extract, transform, load) processes to consolidate data from various sources.

    2. To ensure consistency, standardize the data formats using schema matching and data mapping techniques.

  4. Structural errors:

    1. Meticulously check structural errors, such as incorrect data formats, using data profiling tools.

    2. Then apply data transformation techniques, including parsing and reformatting, to rectify these errors and ensure data consistency.

  5. Outliers:

    1. Identify outliers using statistical methods like z-score and IQR (interquartile range) analysis, or ...