gcp_auth

application_default_credentials.json from GCP SDK installation

project_id

This course is about data engineering using GCP, Python, Kafka, PySpark, and Airflow, covering the entire data lifecycle from ingestion to transformation.

Dock.tar.gz

BigQuery

Airflow

Airflow Custom

Dagster

Airflow-copy

Airflow-custom-copy

Data engineering is currently one of the most in-demand fields in data and technology. It intersects software engineering, DataOps, data architecture, data management, and security. Data engineers, such as analysts and data scientists, lay the foundation to serve data for consumers. 

In this course, you will learn the foundation of data engineering, covering different parts of the entire data life cycle: data warehouse, ingestion, transformation, orchestration, etc. You will also gain hands-on experience building data pipelines using different techniques such as Python, Kafka, PySpark, Airflow, dbt, and more. 

By the end of this course, you will have a holistic understanding of data engineering and be able to build your data pipelines to serve data for various consumers.

Data Engineering Foundations in Python

Data orchestration is the process of transforming data from one format to another.

Data orchestration is an automated process of combining and organizing data from multiple sources, making it ready for data consumers at the right time, in the right order, and in the right way.  

Data orchestration is the process of managing metadata associated with data assets.  

Data orchestration is the process of integrating data from multiple sources into a single repository.

Which statement best describes data orchestration?

Which statement best describes data orchestration?

Which option is *not* the main task of data orchestration?

Which option is not the main task of data orchestration?

In a DAG, every arrow points in one direction.

Which statement is false for a DAG?

To define the tasks that need to be performed in a data workflow

To monitor the performance of data workflows

What is the role of an Airflow operator?

Which Airflow operator is used to check the status of another DAG?

Which Airflow operator is used to check the status of another DAG?

It doesn't handle dependencies between models.

It refers to a dependent model using its physical table name such as `my_database.my_schema.my_table`.

It refers to a dependent model using `ref` such as `ref(my_table)`.

It handles dependencies in a separate configuration file.

How does dbt handle dependencies between models?

How does dbt handle dependencies between models?

Which one is *not* a dbt materialization type?

Which one is not a dbt materialization type?

Apply a full refresh to all tables in the database. 

Migrate data from one database to another.

Create a new database schema from scratch.

Incrementally update only the changed data in the target table.

What does dbt incremental model do?

Test your knowledge of data orchestration.

Getting Started

Data Team Structure

Data Engineering Life Cycle

Cloud Data Architecture

Data Ingestion

Data Modeling

Data Orchestration

Data Quality

Build an End-to-End Data Pipeline for Formula 1 Analysis

Epilogue

Appendix

Quiz: Data Orchestration