Search⌘ K
Home
Courses
Data Engineering Foundations in Python

Challenge: Build an FXRate Reporting Pipeline with Airflow

Explore how to build a foreign exchange rate reporting pipeline using Airflow. Learn to fetch and ingest data from the Frankfurter API, perform timely analysis to calculate daily average rates, and save results in BigQuery tables. This lesson helps you implement data orchestration and manage API and cloud connections within Airflow workflows.

We'll cover the following...

Problem statement

In this challenge, we will create a foreign exchange rate reporting pipeline in Airflow. The pipeline fetches the EUR-to-USD exchange rate of the current day from Frankfurter API every minute from Monday to Friday and ingests it into a BigQuery table. Every five minutes from Monday to Friday, an analysis is performed which calculates the average rate of the current day so far and saves the result into another BigQuery table. The latest analysis of the day will overwrite any previous analysis conducted on the same day.

Note:

  • For the sack of quick testing, we make the job frequency to every minute and every five minutes. But the rate normally doesn't change within five minutes.

  • It's an open question, so there is no definite answer. We should encourage ourselves to come up with different solutions and think about their pros and cons.

Sample data from Frankfurter API

To give us an idea of what ...