HomeCoursesData Engineering Foundations in Python

4.5

Beginner

Updated 3 months ago

Data Engineering Foundations in Python

Gain insights into data engineering foundations, explore data life cycle stages, and delve into creating data pipelines using Python, Kafka, PySpark, Airflow, and dbt.

Join 2.7 million developers at

Overview

Content

Reviews

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataO...Show More

WHAT YOU'LL LEARN

An understanding of the data engineering life cycle

Familiarity with the cloud data warehouse and data modeling techniques

Hands-on experience with data engineering tools such as GCP, Airflow, Spark, and dbt

The ability to build data pipelines from scratch in Python

An understanding of the data engineering life cycle

TAKEAWAY SKILLS

Python

SQL

Content

46 Lessons1 Project7 Quizzes

Getting Started

1 Lessons

Get familiar with essential data engineering skills, including cloud architectures and pipeline orchestration.

Introduction

Data Team Structure

3 Lessons

Get started with the essentials of data team roles and effective team structures.

Data Team Roles

Data Team Structure: Embedded or Centralized?

Quiz: Data Team Structure

Data Engineering Life Cycle

5 Lessons

Master the steps to implement and manage data engineering life cycle stages using Google Cloud.

Set Up the Google Cloud

Introduction to Data Engineering Life Cycle

Ingestion, Transformation, and Visualization

Storage and Infrastructure

Quiz: Data Engineering Life Cycle

Cloud Data Architecture

6 Lessons

Grasp the fundamentals of cloud data evolution, service models, architectures, and best practices for efficiency and security.

Evolution of Cloud Data Architecture

Cloud Service Models

Cloud Data Architectures: Lambda, Kappa, and Data Warehouse

Cloud Data Architectures: Data Lake and Data Mesh

Patterns of Good Cloud Data Architecture

Quiz: Cloud Data Architecture

Data Ingestion

9 Lessons

Take a closer look at methods and tools for efficient data ingestion using Python.

Batch Ingestion

Stream Ingestion

Push vs. Pull

Ingestion Methods—SFTP, API, and Object Storage

Ingestion Methods—CDC

Ingestion Methods—Streaming Platform

Ingest with pandas

Ingest with PySpark

Quiz: Data Ingestion

Data Modeling

8 Lessons

Focus on mastering data modeling and SQL for effective database and analytical operations.

Introduction to Data Modeling

Kimball’s Dimensional Modeling

Steps of Dimensional Modeling

DDL vs. DML

Query Life Cycle

Challenge: BigQuery SQL

Solution: BigQuery SQL

Quiz: Data Modeling

Data Orchestration

8 Lessons

Master data orchestration using tools like Airflow, Dagster, and dbt to enhance workflows.

What Is Data Orchestration?

Airflow Basics

Airflow DAG Design

Challenge: Build an FXRate Reporting Pipeline with Airflow

Solution: Build an FXRate Reporting Pipeline with Airflow

Orchestration Tool: Dagster

Orchestration Tool: dbt

Quiz: Data Orchestration

Mastering Airflow: Building an ETL Pipeline

Project

Data Quality

4 Lessons

Solve problems in measuring and maintaining data quality with schema validation and testing.

Data Quality Measurement

Data Schema: Avro and Protobuf

Manage Data Quality in dbt

Quiz: Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Project

Epilogue

1 Lessons

Look at the fundamentals of data engineering, future paths, and mastering essential tools.

The Way Forward

10.

Appendix

1 Lessons

Explore additional GCP resources and billing management for efficient cloud utilization.

Additional Resources

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Course Author:

Xiaoxu Gao

Developed by MAANG Engineers

Every Educative resource is designed by our team of ex-MAANG software engineers and PhD computer science educators — subject matter experts who’ve shipped production code at scale and taught the theory behind it. The goal is to get you hands-on with the skills you need to stay ahead in today's constantly evolving tech landscape. No videos, no fluff — just interactive, project-based learning with personalized feedback that adapts to your goals and experience.

Trusted by 2.7 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Software Developer

Carlos Matias La Borde

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Personalized Interview Prep

Skip the LeetCode grind with a custom roadmap that adapts to your goals. Hands-on practice for Coding Interviews, System Design, and more.

Mock Interviews

Test your skills in a simulated interview setting. Receive personalized feedback based on your performance. Available for Coding Interviews, System Design, and more.

AI Prompt

Build prompt engineering skills. Practice implementing AI-informed solutions.

Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Course

Mastering Apache Kafka

Learn about Apache Kafka's architecture, client APIs, stream processing, and connectors. Build real-time data streaming and event-driven applications while exploring Kafka's ecosystem.

7 h

intermediate

Course

Data Science in Production: Building Scalable Model Pipelines

Gain insights into building scalable data and model pipelines, explore different cloud environments, delve into streaming workflows, and discover essential tools for creating real-time data products.

8 h

advanced

Course

Cracking the Google Associate Cloud Engineer Certification

Become a cloud pro with strategic GCP skills. Get hands-on experience, understand "Why GCP?", and prepare with a practice exam. No setup, no cleanup, no hassle.

4 h

beginner

Course

Introduction to Big Data and Hadoop

Delve into Big Data essentials, explore data types, and gain insights into Hadoop components like YARN, MapReduce, HDFS, and Spark. Discover foundations to excel in the growing Big Data field.

10 h

beginner

Course

An Introduction to Spark

Gain insights into Spark, its architecture, application lifecycle, and APIs. Delve into data frames, datasets, and Spark SQL to effectively manage and query big data.

2 h 50 m

beginner

Course

From Pandas to PySpark DataFrame

Gain insights into enhancing Python data processing with PySpark. Delve into reading, transforming, aggregating data, and creating user-defined functions, boosting efficiency with Apache Spark.

3 h 3 m

intermediate

Course

Mastering Apache Kafka

intermediate

7 hour

Course

Data Science in Production: Building Scalable Model Pipelines

advanced

8 hour

Course

Cracking the Google Associate Cloud Engineer Certification