Transferring Data with ETL/

...

/

ETL Transformation Example: Sorting and Finalizing the Data

ETL Transformation Example: Sorting and Finalizing the Data

Learn how to sort a CSV file using Bash and complete the last part in the transform stage in the pipeline.

We'll cover the following...

Task 5: Sort the data

Finally, the last task is to sort the data by date in descending order.

Press + to interact
#!/bin/bash
echo -e "\nTask #5 - Sorting the Data"
sort_data(){
    # Sort the data by date
    cat raw_data.csv | sort -t ","  -k1.7,1.10 -k1.1,1.2 -k1.4,1.5 -r >> clean_data.csv
    echo "Done"
}
sort_data

We’ve created a file called transform_data_5_sort.sh to perform these operations. Let’s walk through the code. ...