Search⌘ K
AI Features
Home
Courses
Data Science in Production: Building Scalable Model Pipelines

- BigQuery to Pandas

Explore how to automate data workflows by connecting Python to BigQuery using the Google Cloud library. Learn to set up credentials, query data, and load results into Pandas dataframes, enhancing your ability to build scalable data and model pipelines efficiently.

We'll cover the following...

Workflows automation

One of the ways to automate workflows authored in Python is to directly connect to data sources.

For databases, you can use connectors based on JDBC or native connectors, such as the bigquery module provided by the Google Cloud library. This connector enables Python applications to send queries to BigQuery and load the results as a Pandas dataframe. This process involves setting up a GCP project, installing the prerequisite Python libraries, setting up the Google Cloud command line tools, creating GCP credentials, and finally sending queries to BigQuery programmatically.

If you do not already have a GCP account set up, you’ll need to create a new account. Google provides a $300 credit for getting up and running with the platform.

In our pre-configured ...