Pandas is a popular Python library used to manipulate data, but it has certain limitations in its ability to process large datasets. The Apache Spark analytics library offers significant performance improvements.
This course will help improve your Python-based data processing by leveraging Apache Spark’s multithreading capabilities through the PySpark library. You’ll start by reading data into a PySpark DataFrame before performing basic input/output functions, such as renaming attributes, selecting, and writing data. You’ll move onto transformation functions like aggregation, statistical analysis, and joins before creating custom, user-defined functions. At each step, you’ll get a quick Pandas review before being walked through leveraging the more robust PySpark library to unlock Apache Spark.
By the end of this course, you’ll be able to quickly and reliably process large amounts of data, even stored across multiple files, using PySpark.
Pandas is a popular Python library used to manipulate data, but it has certain limitations in its ability to process large datas...Show More
WHAT YOU'LL LEARN
A working knowledge of Apache Spark and the PySpark library for Python
A strong understanding of the advantages of using PySpark instead of Pandas for processing large datasets
The ability to calculate some Metrics or produce aggregated analytics reporting solutions
The ability to write Production Code in PySpark
A working knowledge of Apache Spark and the PySpark library for Python
Show more
Content
1.
Introduction
2 Lessons
Learn how to use PySpark for large-scale data processing and Amazon Review Data analysis.
2.
Data Input/Output
10 Lessons
Walk through data input/output processes including reading, renaming, selecting, saving, and challenges.
3.
Data Transformation
16 Lessons
Work your way through transforming data, handling date-time, imputing, and evaluating reviews using pandas and PySpark.
4.
User Defined Function (UDF)
8 Lessons
Build a foundation in creating and using UDFs in PySpark for custom transformations.
6.
Appendix
2 Lessons
Focus on the Amazon Review Data (2018) and Pandas vs. PySpark performance.
Certificate of Completion
Showcase your accomplishment by sharing your certificate of completion.
Course Author:
Developed by MAANG Engineers
Trusted by 2.8 million developers working at companies
"These are high-quality courses. Trust me the price is worth it for the content quality. Educative came at the right time in my career. I'm understanding topics better than with any book or online video tutorial I've done. Truly made for developers. Thanks"
Anthony Walker
@_webarchitect_
"Just finished my first full #ML course: Machine learning for Software Engineers from Educative, Inc. ... Highly recommend!"
Evan Dunbar
ML Engineer
"You guys are the gold standard of crash-courses... Narrow enough that it doesn't need years of study or a full blown book to get the gist, but broad enough that an afternoon of Googling doesn't cut it."
Software Developer
Carlos Matias La Borde
"I spend my days and nights on Educative. It is indispensable. It is such a unique and reader-friendly site"
Souvik Kundu
Front-end Developer
"Your courses are simply awesome, the depth they go into and the breadth of coverage is so good that I don't have to refer to 10 different websites looking for interview topics and content."
Vinay Krishnaiah
Software Developer
Hands-on Learning Powered by AI
See how Educative uses AI to make your learning more immersive than ever before.
AI Prompt
Code Feedback
Explain with AI
AI Code Mentor
Free Resources
