Know Your Data
Learn where data comes from, what types it takes, and how it’s stored around us.
We'll cover the following...
As a data engineer, everything starts with understanding the data itself. Where is it coming from? What form does it take—structured, unstructured, or something in between? How often does it change? These core questions may sound simple, but they define how you'll collect, store, and manage data across systems. Before any tools or tech come into play, it's this clarity about your data that sets the stage for building efficient, reliable data workflows.
In data engineering, designing the wrong data structure due to a poor understanding of the source data is a top reason why pipelines fail at scale.
Know the backstory
Every dataset has a backstory. It might be clean and well-organized, or it might be messy and inconsistent. Maybe it was collected through a web form, a sensor, or a survey. Each of these origins shapes what the data can tell us—and what it can’t.
Let’s head back to your kitchen. Remember all those scattered recipes—some scribbled on napkins, a few saved in your notes app, others stuck to the fridge? Last time, we talked about organizing them neatly with labels, categories, and a proper system—that’s what a database helped us do. But here’s the next step: before you decide how to store or organize anything, you need to understand what you’re dealing with. ...