Transferring Data with ETL - AI-Powered Course

Transferring Data with ETL

This course covers creating and orchestrating ETL pipelines using the industry’s best practices and tools: Python, SQL, Apache Spark, and Apache Airflow.

Intermediate

57 Lessons

10h

Certificate of Completion

This course covers creating and orchestrating ETL pipelines using the industry’s best practices and tools: Python, SQL, Apache Spark, and Apache Airflow.

AI-POWERED

Explanations

AI-POWERED

Explanations

This course includes

1 Project

133 Playgrounds

7 Quizzes

This course includes

1 Project

133 Playgrounds

7 Quizzes

Course Overview

ETL stands for extract, transform, and load. It’s a collection of processes that combine data from various sources and load them into data warehouses or other data repositories. ETL is crucial for providing data used for business intelligence and analytics. In this course, you’ll experiment with extracting data from various database solutions such as MySQL, PostgreSQL, and MongoDB. You’ll use query and scripting languages like SQL, Python, and Apache Spark to process data and load it to data repositories o...Show More

What You'll Learn

An understanding of the extract, transform, and load steps in an ETL pipeline

Hands-on experience implementing and orchestrating the ETL pipelines

Understanding of databases, data warehousing, data processing, and data ingestion

Hands-on experience with ETL tools such as Python, SQL, Apache Spark, and Apache Airflow

What You'll Learn

An understanding of the extract, transform, and load steps in an ETL pipeline

Course Content

Introduction

In this chapter, you’ll explore ETL pipelines, covering extraction, transformation, loading, batch vs. stream processing, and data warehousing.

Getting Started

ETL Pipeline Stages

What Is an ETL Pipeline?A New Paradigm—ELT ETL Example—Extraction ETL Transformation Example: Addressing Data Quality Issue ETL Transformation Example: Handling Missing Values and Data ETL Transformation Example: Sorting and Finalizing the Data ETL Example—Load ETL Example—Scheduling Batch vs. Stream Processing Data Warehouse Examples and Use Cases Quiz: ETL Pipelines

E: Extract

This chapter uses practical exercises to cover data extraction methods like web scraping, REST APIs, MySQL, PostgreSQL, and Google BigQuery.

Introduction Data Extraction Methods Overview Extracting Data with Web Scraping Web Scraping Exercise: Reading the Data Web Scraping Exercise: DataFrames to CSV Extraction Using a REST API Exercise: Extracting Data with a REST API Full Extraction From MySQL Database Incremental Extraction From MySQL Database Extraction From MySQL’s Binary Log Extract From PostgreSQL Database Extraction From Google BigQuery Exercise: Databases Quiz: Extracting Data

T: Transform

This chapter covers data typing, anonymizing, cleaning with Python and Spark, and SQL techniques for transforming data efficiently.

Introduction Data Typing and Structuring Using Python Exercise: Data Typing and Structuring Anonymizing and Encrypting Using Python Exercise: Anonymizing and Encrypting the Data Data Cleaning Using Apache Spark: Missing Data Data Cleaning Using Apache Spark: Duplicate Data Exercise: Data Cleaning Filtering, Sorting, Aggregating, and Binning Using SQL Exercise: Filtering, Sorting, Aggregating, and Binning Quiz: Transforming Data

L: Load

This chapter explores hosting options, data loading methods, and the COPY command for efficient data handling.

Introduction Hosting and Deployment: On-Premise vs. Cloud-Based Hosting and Deployment: Open-Sourced vs. Proprietary Loading Methods: Scheduled vs. On-Demand Loading Methods: Full Loading vs. Incremental Loading Loading Data Using the COPY Command Exercise: The COPY Command Exercise: Incremental Load Quiz: Loading Data

Orchestration

In this chapter, you’ll explore deploying Airflow, building ETL pipelines, and managing extraction, transformation, and loading tasks.

Introduction Deploying Airflow ETL Pipeline Exercise: Extracting Data ETL Pipeline Example: Airflow Extraction Task ETL Pipeline Exercise: Transform ETL Pipeline Example: Airflow Transform Task ETL Pipeline Example: Load Interacting with Airflow

ETL Pipeline: Fraud Detection Preprocessing

Project

Conclusion

1 Lesson

This chapter summarizes ETL pipelines, covering fundamental concepts, real-world applications, and tools like Apache Airflow.

Trusted by 1.4 million developers working at companies

Anthony Walker

@_webarchitect_

Evan Dunbar

ML Engineer

Carlos Matias La Borde

Software Developer

Souvik Kundu

Front-end Developer

Vinay Krishnaiah

Software Developer

Eric Downs

Musician/Entrepeneur

Kenan Eyvazov

DevOps Engineer

Souvik Kundu

Front-end Developer

Eric Downs

Musician/Entrepeneur

Anthony Walker

@_webarchitect_

Evan Dunbar

ML Engineer

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI-Powered Mock Interviews

Put your skills to the test in a simulated interview setting. Receive personalized feedback based on your performance. Available in Premium & Premium Plus plans.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath