INTERACTIVE COURSE

From Pandas to PySpark DataFrame

Intermediate

39 Lessons

3h 3min

Certificate of Completion

AI Explanations

54 Playgrounds

27 Illustrations

Takeaway Skills

A working knowledge of Apache Spark and the PySpark library for Python

A strong understanding of the advantages of using PySpark instead of Pandas for processing large datasets

The ability to calculate some Metrics or produce aggregated analytics reporting solutions

The ability to write Production Code in PySpark

Course Overview

Pandas is a popular Python library used to manipulate data, but it has certain limitations in its ability to process large datasets. The Apache Spark analytics library offers significant performance improvements. This course will help improve your Python-based data processing by leveraging Apache Spark’s multithreading capabilities through the PySpark library. You’ll start by reading data into a PySpark DataFrame before performing basic input/output functions, such as renaming attributes, selecting, and wr...Show More

Course Content

Introduction

Getting Started

Overview of Dataset

Data Input/Output

Introduction to Data Input and Output Read Data into DataFrame Rename Attributes Select a Subset of Attributes Data Input and Output: Save a Snapshot Read Parquet Data Source Write Production Code Quiz: Data Input and Output Challenge: Data Input and Output Solution: Data Input and Output

Data Transformation

Introduction to Data Transformation Setup Handling Date-time Impute Unavailable Data Points Average Review per Product Total Number of Reviews for Each Product Distribution of the Review Text Length Yearly Median Review Top reviews of 2017 Compare Total Review of 2016 and 2017 Conversion Between Wide and Long Format using melt and pivot Date Transformation: Save a Snapshot Avoid Global Scope Quiz: Data Transformation Challenge: Data Transformation Solution: Data Transformation

User Defined Function (UDF)

Introduction to User-defined Functions Object Conversion Between Python and Scala Writing UDF UDF in Action UDF: Save a snapshot Quiz: User-defined Functions Challenge: User-defined Functions Solution: User Defined Function

Wrapping Up

Conclusion

Appendix

2 Lessons

COURSE AUTHOR

How You'll Learn

You don’t get better at swimming by watching others. Coding is no different. Practice as you learn with live code environments inside your browser.

Videos are holding you back. Educative‘s interactive, text-based lessons accelerate learning — no setup, downloads, or alt-tabbing required.

Learn faster and smarter with adaptive AI tools embedded in every Educative course.

Built-in assessments let you test your skills. Completion certificates let you show them off.

FOR TEAMS

Interested in this course for your business or team?

Unlock this course (and 1,000+ more) for your entire org with DevPath