Extraction From Google BigQuery

Data engineers, data scientists, and developers must know how to extract data from cloud services. Using cloud services, companies today can scale their workload dynamically, attain global accessibility to their application for users worldwide, and acquire mature data products with great security, support, and reliability without deploying and managing them on their infrastructure.

Google has a suite of cloud computing services available on its platform. These services allow organizations and individuals to store, process, and access data and applications on Google’s highly scalable and reliable infrastructure. One example of these services is Google BigQuery.

BigQuery is Google’s data warehouse solution. It lets us analyze and query petabytes scale datasets using SQL syntax. BigQuery can be used for storage and analytics without worrying about managing the infrastructure and resources. We can use BigQuery using Google’s console, the command line, or by using Python. Let’s look at how to connect to BigQuery using Python and extract data.

Acquiring data

Datasets in BigQuery are like databases in MySQL or schemas in PostgreSQL, and they’re a placeholder for multiple tables. We can easily create a dataset using BigQuery’s platform or use one of their public datasets.

We’ll use one of BigQuery’s public dataset called google_trends for this demonstration. We’ll query a table within this dataset called top_rising_terms which showcases the top trending terms according to Google Trends.

Extracting data

Next, to connect Python to BigQuery, we need to authenticate ourselves with Google. The recommended way of doing it is by creating a service account on Google’s platform and downloading the credentials as a JSON file. Once we have the credentials, we should create an environment variable called GOOGLE_APPLICATION_CREDENTIALS and set its value to be the path of the JSON file.

After that, we can create a BigQuery client and query the data. Click the “Run” button in the following code widget to see the top rising terms over the last month according to Google Trends.