Data Engineering Fundamentals - Data Engineering Foundations in Python

Data Engineering Foundations in Python

0%

1.Getting Started

2.Data Team Structure

Data Team Roles Data Team Structure: Embedded or Centralized?Quiz: Data Team Structure

3.Data Engineering Life Cycle

Set Up the Google Cloud Introduction to Data Engineering Life Cycle Ingestion, Transformation, and Visualization Storage and Infrastructure Quiz: Data Engineering Life Cycle

4.Cloud Data Architecture

Evolution of Cloud Data Architecture Cloud Service Models Cloud Data Architectures: Lambda, Kappa, and Data Warehouse Cloud Data Architectures: Data Lake and Data Mesh Patterns of Good Cloud Data Architecture Quiz: Cloud Data Architecture

5.Data Ingestion

Batch Ingestion Stream Ingestion Push vs. Pull Ingestion Methods—SFTP, API, and Object Storage Ingestion Methods—CDC Ingestion Methods—Streaming Platform Ingest with pandas Ingest with PySpark Quiz: Data Ingestion

6.Data Modeling

Introduction to Data Modeling Kimball’s Dimensional Modeling Steps of Dimensional Modeling DDL vs. DML Query Life Cycle Challenge: BigQuery SQL Solution: BigQuery SQL Quiz: Data Modeling

7.Data Orchestration

What Is Data Orchestration?Airflow Basics Airflow DAG Design Challenge: Build an FXRate Reporting Pipeline with Airflow Solution: Build an FXRate Reporting Pipeline with Airflow Orchestration Tool: Dagster Orchestration Tool: dbt Quiz: Data Orchestration

Project

Mastering Airflow: Building an ETL Pipeline

8.Data Quality

Data Quality Measurement Data Schema: Avro and Protobuf Manage Data Quality in dbt Quiz: Data Quality

Mini Project

Build an End-to-End Data Pipeline for Formula 1 Analysis

9.Epilogue

The Way Forward

10.Appendix

Additional Resources

Mock Interview

Data Engineering Fundamentals