Search⌘ K
AI Features
Home
Courses
Transferring Data with ETL

ETL Transformation Example: Sorting and Finalizing the Data

Explore how to execute the transformation step within an ETL pipeline by sorting raw data by date in descending order. Learn to implement a shell script that cleans and formats the data for subsequent loading, ensuring the dataset meets analytic needs before moving forward.

We'll cover the following...

Task 5: Sort the data

Finally, the last task is to sort the data by date in descending order.

Shell
#!/bin/bash
echo -e "\nTask #5 - Sorting the Data"
sort_data(){
    # Sort the data by date
    cat raw_data.csv | sort -t ","  -k1.7,1.10 -k1.1,1.2 -k1.4,1.5 -r >> clean_data.csv
    echo "Done"
}
sort_data

We’ve created a file called transform_data_5_sort.sh to perform these operations. Let’s walk through the code. ...