Navigate Data Wrangling

As data analysts, we might wish for clean, consistent, and ready-to-explore datasets. But more often than not, we face a very different reality. Columns may be misnamed or missing, values duplicated or out of range, formats inconsistent across files, and data scattered across various files and sources.

Before we can explore trends, create visualizations, or run any analysis, we need to make sense of the chaos in our data. This is where data wrangling comes in. It’s the essential process of turning raw, messy data into clean, structured, and usable information.

Data wrangling

Data wrangling is the process of cleaning, structuring, and enriching raw data into a desired format for better decision-making. It involves taking messy, unorganized, or incomplete datasets and transforming them into something usable. Whether we are removing duplicate entries, fixing incorrect values, merging datasets, or creating new calculated fields, all of it falls under the broad umbrella of data wrangling.

The goal of data wrangling isn’t just about cleaning up data for the sake of neatness. It’s about preparing data so that we can extract reliable insights, build meaningful charts, and support sound decision-making.

Data wrangling workflow

Think of the data wrangling workflow as a reliable guide for preparing any dataset, from disorganized data to analysis-ready. Let’s break it down into a clear, step-by-step process:

Each step builds on the last, and skipping any one can cause problems later on. Let’s explore what happens at each stage and why it matters.

1. Ingest: Collecting the raw materials

Before we can clean or analyze anything, we need to gather raw data, and that’s what ingestion is all about. This step brings data together from different sources into one place, where we can begin working with it.

Note: We’ve already explored data collection in the previous chapter, so some of this may feel familiar. Here, we’re taking it a step further: bringing that collected data into our analysis environment so we can start preparing it.

Data ingestion can include:

• Gathering records from structured data sources.

• Loading files like CSVs or Excel sheets from local or shared storage.

• Accessing data directly from web sources.

• Importing semi-structured formats like XML.

• Bringing data from various cloud storage locations.

The goal is to gather all relevant data, no matter the format or location, into our working environment for analysis.

Example

Let’s say we’re analyzing customer purchases. We might start by gathering order data from a central repository. Next, we download product details from a CSV file maintained by the inventory team. To get the latest prices, we access live pricing data from an online source. Once we’ve gathered all three sources, we combine them into a single coherent dataset. Only after this ingestion step can we begin to clean, merge, and make sense of the data.

Challenges

While ingestion may sound straightforward, it often comes with technical and logistical hurdles that we need to plan for.

• Different formats: Data might come from structured sources, CSV files, or web-based data feeds, each needing its own handling and alignment.

• Access issues: Some sources require credentials, logins, or specific permissions to connect and fetch data.

• Big data loads: Massive datasets can overwhelm memory or crash tools. Processing data in smaller chunks can help manage it safely.

• Real-time delays: Live data (like from sensors or web feeds) may be incomplete, late, or out of order, requiring careful handling and timestamp syncing.

2. Clean: Making the data usable

Cleaning is the step where we make the raw, ingested data trustworthy. This means correcting errors, removing duplicates, handling missing values, and fixing formatting issues. ...