ETL Example—Extraction
Explore how to build a simple ETL pipeline using Bash shell scripting. Learn to extract raw lottery data from a CSV file online, identify data issues, and prepare for transformation and loading to a PostgreSQL database. This lesson emphasizes choosing practical tools and handling external data sources efficiently.
We'll cover the following...
ETL pipelines should only be as complicated as they need to be. It's easy to get swept away by the latest industry trends and feel overwhelmed with new software and tools. When building an ETL pipeline, we’ll need to choose the most appropriate tool for each step. But not all pipelines have to be complex.
To demonstrate this, let’s build an entire ETL pipeline from scratch using the shell scripting language Bash. This will provide a valuable example of an ETL pipeline and show that it’s possible to do it (efficiently) using even a simple tool like Bash.
Bash is a Unix shell scripting language released in 1989. It stands for ...