Shape It Right
Learn to reshape and prepare structured data for pipeline-friendly processing with pandas.
Data engineers do more than just store and move data—they design systems that turn raw information into meaningful insight. Imagine walking into a server room where every cable is a tangled mess. That’s what messy data looks like. We need to reshape and organize it before any analysis or downstream tasks can even begin.
Think of your raw dataset like a container full of LEGO pieces. If you want to build something meaningful—a chart, a report, or a data model—you’ll need the right bricks in the right places. That’s what tidy data gives us: clean rows, clear columns, and well-structured components.
Reshaping data is not just about formatting—it’s about making data usable. In this lesson, we’ll unpack what tidy means, explore the difference between wide and long formats, and master reshaping tools in pandas—melt(), pivot(), pivot_table(), stack(), and unstack()—so we can transform DataFrame to fit the task at hand.
What is tidy data?
Tidy data follows three simple rules:
Each variable forms a column. Every distinct attribute or measurement is stored in its own column.
Each observation forms a row. Each row represents one complete set of measurements or attributes for a single entity or event.
Each type of observational unit forms a table. Different entities or observational types should be stored in separate tables to avoid mixing unrelated data.
This consistent and predictable structure is essential because many data manipulation and visualization tools expect data to be tidy. When data is tidy, we can easily apply filters, groupings, summaries, and charts without requiring complicated reshaping.
Fun fact: The term "tidy data" was popularized by statistician Hadley Wickham. It’s now a fundamental principle in modern data handling—and a must-know for every data engineer.
Wide format vs. long format
Understanding how our data is structured helps us decide how to reshape it.
Wide format: In wide format, similar measurements are spread across multiple columns. This formal is useful for quick human inspection, but is harder to automate.
Note: For example, monthly sales might be stored in separate columns like Jan_Sales, Feb_Sales, Mar_Sales. This makes it easy to compare values side-by-side, but may create challenges for functions or visualizations that require data in a stacked format. Let’s create a sample dataset and see what a wide format looks like:
import pandas as pd# Wide format exampledf_wide = pd.DataFrame({'Product': ['A', 'B', 'C'],'Jan_Sales': [100, 150, 200],'Feb_Sales': [110, 140, 210],'Mar_Sales': [120, 160, 220]})print("Wide format data:")print(df_wide)
Here, sales for each month are separate columns—this ...