gcp_auth

application_default_credentials.json from GCP SDK installation

project_id

This course is about data engineering using GCP, Python, Kafka, PySpark, and Airflow, covering the entire data lifecycle from ingestion to transformation.

Dock.tar.gz

BigQuery

Airflow

Airflow Custom

Dagster

Airflow-copy

Airflow-custom-copy

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. 

In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. 

By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data Engineering Foundations in Python

Bridging the gap between data engineers and data analysts/scientists

What's the main role of an analytics engineer?

What’s the main role of an analytics engineer?

A strong relationship between engineers and business

Consistent best data practices across the organization

An environment to easily share both technical and functional knowledge

What contributes to a good data team?

**(True or False)** A centralized team structure is better than an embedded team structure.

(True or False) A centralized team structure is better than an embedded team structure.

Easily sharing technical and functional knowledge with other DE/DA/DS

Consistent best data practices and vision across the organization

Feasible for early start-ups with not enough engineers

Strong relationship between engineering and business teams

What is *not* an advantage of the centralized team structure?

What is not an advantage of the centralized team structure?

**(True or False)** Software engineers are one of the main data consumers.

(True or False) Software engineers are one of the main data consumers.

Test your knowledge of data team structures.

Getting Started

Data Team Structure

Data Engineering Life Cycle

Cloud Data Architecture

Data Ingestion

Data Modeling

Data Orchestration

Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Epilogue

Appendix

Quiz: Data Team Structure