Filtering and Aggregating Data with Transform Streams

Filtering and aggregating data with Transform streams

Transform streams are the perfect building blocks for implementing data transformation pipelines. In the previous section, we illustrated an example of a Transform stream that can replace words in a stream of text. But Transform streams can be used to implement other types of data transformation as well. For instance, it’s really common to use Transform streams to implement data filtering and data aggregation.

Just to make a practical example, let’s imagine we are asked by a Fortune 500 company to analyze a big file containing all the sales for the year 2020. The company wants us to use data.csv, a sales report in CSV format, to calculate the total profit for the sales made in Italy.

For simplicity, let’s imagine the sales data that’s stored in the CSV file contains three fields per line: item type (type), country of sale (country), and profit (profit). Such a file could look like this: