Mastering Big Data with PySpark
Gain insights into PySpark within big data. Learn about data ingestion, distributed computing, data processing, and performance optimization to solve real-world problems and apply machine learning.
After completing my Bachelor’s and Master’s degrees in Agriculture in India, I pursued my Ph.D. in crop genetics from the University of Nottingham, UK, where I laid the foundations in molecular biology. Intrigued by the potential of next-generation sequencing, I delved into the field of Bioinformatics, joining as a Postdoctoral researcher in Dr. Julin Maloof’s lab at the University of California, Davis. Here, I focused on developing genomic resources for Brassica rapa, further fueling my interest in computational biology. Transitioning fully into computational work, I joined CyVerse as a Science Analyst, where I developed innovative computational processes, integrating tools and algorithms into CyVerse’s infrastructure. I bring a unique perspective to the field with a diverse background in Genomics and bioinformatics and a specialization in applying predictive analytics across various biological problems. Currently serving as the head of Data Science at Greenlight Biosciences, I lead a team of Data Scientists and Bioinformatics Scientists to leverage cutting-edge technology and machine learning algorithms to address complex biological challenges. Our focus on RNA and dsRNA enables us to contribute significantly to vaccine, biopesticide, and biofungicide development. Outside of work, I enjoy playing badminton, watching sports, and spending time with friends. I am also passionate about teaching and knowledge sharing. As a certified Data Science instructor at Carpentries and DataCamp, I share my expertise in Data Science, programming, and software technologies such as Git, Docker, and Singularity. Additionally, I authored a book on Deep Learning for Genomics and developed a popular course on PySpark on DataCamp. My ultimate goal is to empower others with the tools and skills needed to tackle the most challenging problems in the big data ecosystem.
Join 3.1 million developers at
What You'll Learn
- An understanding of the big data ecosystem, including data ingestion, integration methods, and big data storage options
- A working knowledge of distributed computing fundamentals, covering parallel processing, partitioning strategies, and load balancing methodologies
- The ability to utilize PySpark for diverse data operations, including processing, transformation, and analysis
- Familiarity with basic and advanced data types, Spark SQL, machine learning algorithms, and data mining within PySpark
- A working knowledge of PySpark's integration capabilities with various big data tools, such as Hadoop, Kafka, Hive, and others
Learning Roadmap
1.
Introduction to the Course
Introduction to the Course
2.
Introduction to Big Data
Introduction to Big Data
3.
Exploring PySpark Core and RDDs
Exploring PySpark Core and RDDs
5 Lessons
5 Lessons
4.
PySpark DataFrames and SQL
PySpark DataFrames and SQL
6 Lessons
6 Lessons
5.
Customer Churn Analysis Using PySpark
Customer Churn Analysis Using PySpark
3 Lessons
3 Lessons
6.
Machine Learning with PySpark
Machine Learning with PySpark
6 Lessons
6 Lessons
7.
Modeling with PySpark MLlib
Modeling with PySpark MLlib
5 Lessons
5 Lessons
8.
Predicting Diabetes in Patients Using PySpark MLlib
Predicting Diabetes in Patients Using PySpark MLlib
3 Lessons
3 Lessons
9.
Performance Optimization in PySpark
Performance Optimization in PySpark
5 Lessons
5 Lessons
10.
PySpark Optimization: Analyzing NYC Restaurants Data
PySpark Optimization: Analyzing NYC Restaurants Data
3 Lessons
3 Lessons
11.
Integrating PySpark with Other Big Data Tools
Integrating PySpark with Other Big Data Tools
4 Lessons
4 Lessons
Upendra Kumar Devisetty
A wet-lab molecular biology scientist turned bioinformatics expert and head of Data Science at Greenlight Biosciences. Author of Deep Learning for Genomics book and Big Data fundamentals via PySpark at Datacamp.
Trusted by 3.1 million developers working at companies
Anthony Walker
@_webarchitect_
Evan Dunbar
ML Engineer
Software Developer
Carlos Matias La Borde
Souvik Kundu
Front-end Developer
Vinay Krishnaiah
Software Developer
Built for 10x Developers












Free Resources