HomeCoursesTransferring Data with ETL

Intermediate

10h

Updated 3 months ago

Transferring Data with ETL
Save

Master ETL processes, explore data extraction from MySQL, PostgreSQL, and MongoDB, and learn about scheduling and automating ETL pipelines using tools like Apache Airflow and Python.

Join 2.7 million developers at

Overview

Content

Reviews

ETL stands for extract, transform, and load. It’s a collection of processes that combine data from various sources and load them into data warehouses or other data repositories. ETL is crucial for providing data used for business intelligence and analytics. In this course, you’ll experiment with extracting data from various database solutions such as MySQL, PostgreSQL, and MongoDB. You’ll use query and scripting languages like SQL, Python, and Apache Spark to process data and load it to data repositories or cloud solutions like Google’s GCP. Finally, you’ll learn how to schedule your ETL pipelines using cronjobs or automate and monitor them using open-source tools like Apache Airflow and Python’s pandas library. After completing this course, you’ll have a strong grasp of various methods, tools, and techniques for transferring data from a source to its destination using ETL pipelines.

ETL stands for extract, transform, and load. It’s a collection of processes that combine data from various sources and load them...Show More

WHAT YOU'LL LEARN

An understanding of the extract, transform, and load steps in an ETL pipeline

Hands-on experience implementing and orchestrating the ETL pipelines

Understanding of databases, data warehousing, data processing, and data ingestion

Hands-on experience with ETL tools such as Python, SQL, Apache Spark, and Apache Airflow

An understanding of the extract, transform, and load steps in an ETL pipeline

Content

57 Lessons1 Project7 Quizzes

Introduction

14 Lessons

Get familiar with building ETL pipelines, their stages, and practical data transformation examples.

Getting Started

ETL Pipeline Stages

What Is an ETL Pipeline?

A New Paradigm—ELT

ETL Example—Extraction

ETL Transformation Example: Addressing Data Quality Issue

ETL Transformation Example: Handling Missing Values and Data

ETL Transformation Example: Sorting and Finalizing the Data

ETL Example—Load

ETL Example—Scheduling

Batch vs. Stream Processing

Data Warehouse

Examples and Use Cases

Quiz: ETL Pipelines

E: Extract

14 Lessons

Get started with techniques for data extraction from various sources including databases, APIs, and web scraping.

Introduction

Data Extraction Methods Overview

Extracting Data with Web Scraping

Web Scraping Exercise: Reading the Data

Web Scraping Exercise: DataFrames to CSV

Extraction Using a REST API

Exercise: Extracting Data with a REST API

Full Extraction From MySQL Database

Incremental Extraction From MySQL Database

Extraction From MySQL’s Binary Log

Extract From PostgreSQL Database

Extraction From Google BigQuery

Exercise: Databases

Quiz: Extracting Data

T: Transform

11 Lessons

Master the steps to transform raw data into usable formats through cleaning, structuring, anonymizing, and aggregating.

Introduction

Data Typing and Structuring Using Python

Exercise: Data Typing and Structuring

Anonymizing and Encrypting Using Python

Exercise: Anonymizing and Encrypting the Data

Data Cleaning Using Apache Spark: Missing Data

Data Cleaning Using Apache Spark: Duplicate Data

Exercise: Data Cleaning

Filtering, Sorting, Aggregating, and Binning Using SQL

Exercise: Filtering, Sorting, Aggregating, and Binning

Quiz: Transforming Data

L: Load

9 Lessons

Grasp the fundamentals of loading transformed data into repositories, hosting options, and loading strategies.

Introduction

Hosting and Deployment: On-Premise vs. Cloud-Based

Hosting and Deployment: Open-Sourced vs. Proprietary

Loading Methods: Scheduled vs. On-Demand

Loading Methods: Full Loading vs. Incremental Loading

Loading Data Using the COPY Command

Exercise: The COPY Command

Exercise: Incremental Load

Quiz: Loading Data

Orchestration

8 Lessons

Take a closer look at orchestrating ETL pipelines with Apache Airflow, deployment, and task management.

Introduction

Deploying Airflow

ETL Pipeline Exercise: Extracting Data

ETL Pipeline Example: Airflow Extraction Task

ETL Pipeline Exercise: Transform

ETL Pipeline Example: Airflow Transform Task

ETL Pipeline Example: Load

Interacting with Airflow

ETL Pipeline: Fraud Detection Preprocessing

Project

Conclusion

1 Lessons

Implement essential ETL skills and tools for effective data architecture and automation.

Final Thoughts

Build a News ETL Data Pipeline Using Python and SQLite

Project

Certificate of Completion

Showcase your accomplishment by sharing your certificate of completion.

Developed by MAANG Engineers

Every Educative resource is designed by our team of ex-MAANG software engineers and PhD computer science educators — subject matter experts who’ve shipped production code at scale and taught the theory behind it. The goal is to get you hands-on with the skills you need to stay ahead in today's constantly evolving tech landscape. No videos, no fluff — just interactive, project-based learning with personalized feedback that adapts to your goals and experience.

Trusted by 2.7 million developers working at companies

"These are high-quality courses. Trust me. I own around 10 and the price is worth it for the content quality. EducativeInc came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"

Anthony Walker

@_webarchitect_

"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"

Evan Dunbar

ML Engineer

"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."

Carlos Matias La Borde

Software Developer

"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"

Souvik Kundu

Front-end Developer

"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."

Vinay Krishnaiah

Software Developer

"I've tried probably 5-7 different sites and Educative is easily the best. It perfectly blends explanation with interactivity"

Eric Downs

Musician/Entrepeneur

Hands-on Learning Powered by AI

See how Educative uses AI to make your learning more immersive than ever before.

Instant Code Feedback

Evaluate and debug your code with the click of a button. Get real-time feedback on test cases, including time and space complexity of your solutions.

AI Mock Interviews

Test your skills in a simulated interview setting. Receive personalized feedback based on your performance. Available for Coding Interviews, System Design, and more.

Adaptive Learning

At various checkpoints throughout Educative courses, you will be prompted to take a quick assessment. Receive a condensed curriculum tailored to your strengths and skill gaps.

Explain with AI

Select any text within any Educative course, and get an instant explanation — without ever leaving your browser.

AI Code Mentor

AI Code Mentor helps you quickly identify errors in your code, learn from your mistakes, and nudge you in the right direction — just like a 1:1 tutor!

Course

Data Science in Production: Building Scalable Model Pipelines

Gain insights into building scalable data and model pipelines, explore different cloud environments, delve into streaming workflows, and discover essential tools for creating real-time data products.

8 h

advanced

Course

Mastering Big Data with PySpark

Gain insights into PySpark within big data. Learn about data ingestion, distributed computing, data processing, and performance optimization to solve real-world problems and apply machine learning.

12 h

beginner

Course

Mastering Big Data with Apache Spark and Java

Gain insights into Spark Java API, learn about data transformations and SQL operations, and discover how to integrate big data and Java for scalable, high-speed processing.

17 h 15 m

intermediate

Course

Getting Started with SQL and Relational Databases

Gain insights into relational databases and SQL. Learn about creating, managing, and editing tables using MySQL. Explore complex queries, indexes, and join functions for efficient data handling.

9 h 40 m

beginner

Course

Predictive Data Analysis with Python

Learn predictive data analysis with Python using NumPy, Pandas, Matplotlib, and Seaborn. Apply skills to real-world finance and advertising projects to extract and visualize insights.

10 h

beginner

Course

Python Data Analysis and Visualization

Delve into Python data analysis and visualization using NumPy, SciPy, Pandas, and Seaborn. Gain insights into reading, cleaning, and visualizing data to generate actionable insights.

2 h 30 m

beginner

Course

Data Science in Production: Building Scalable Model Pipelines

advanced

8 hour

Course

Mastering Big Data with PySpark

beginner

12 hour

Course

Mastering Big Data with Apache Spark and Java