Gain insights into enhancing Python data processing with PySpark. Delve into reading, transforming, aggregating data, and creating user-defined functions, boosting efficiency with Apache Spark.

frompandastopyspark.tar.gz

PandasAndPyspark

jupyter

spa-jupyter

spa-snapshot

spa-copy

spa-copy-copy

spa-copy-copy-5klyxp

spa-snapshot-copy

spa-snapshot-copy-copy

spa-copy-copy-5klyxp-copy

Pandas is a popular Python library used to manipulate data, but it has certain limitations in its ability to process large datasets. The Apache Spark analytics library offers significant performance improvements.

This course will help improve your Python-based data processing by leveraging Apache Spark’s multithreading capabilities through the PySpark library. You’ll start by reading data into a PySpark DataFrame before performing basic input/output functions, such as renaming attributes, selecting, and writing data. You’ll move onto transformation functions like aggregation, statistical analysis, and joins before creating custom, user-defined functions. At each step, you’ll get a quick Pandas review before being walked through leveraging the more robust PySpark library to unlock Apache Spark.

By the end of this course, you’ll be able to quickly and reliably process large amounts of data, even stored across multiple files, using PySpark.

From Pandas to PySpark DataFrame

# Calculate the statistics of the review text length in pandas

In pandas, `applymap` can be used to calculate the length of the review text column. When we calculate the length of text, both frameworks throw a runtime exception if there are `NaN` or `None` values in the column. Therefore, missing values must be filled with inputs before applying any function.


import pandas as pd
PANDAS_DISPLAY_SETTING = [
    ('display.max_rows', 500),
    ('display.max_columns', 500),
    ('display.width', 1000)
]


def set_pd_display(module: pd) -> bool:
    for item in PANDAS_DISPLAY_SETTING:
        module.set_option(*item)
    return True


# pathconfig.py
from pathlib import Path
import time
import yaml

ROOT = Path(__file__).parent
DATA_LAKE = ROOT.joinpath("data")
RAW_DATA = DATA_LAKE.joinpath("raw")
SNAPSHOT = DATA_LAKE.joinpath("snapshot")

PATH_TOY_GAME_DATA = "/Toys_and_Games_5.json"

def create_path_snapshot() -> Path:
    """
    Create path to shapshot wit time stamp
    :return: Path like object
    """
    path_fixed = 'snapshot_{}'.format(int(time.time()))
    return SNAPSHOT.joinpath(path_fixed)


# TODO: Store The Snapshot
def create_path_snapshot_spark():
    path_fixed = 'data/snapshot/pyspark/snapshot_{}'
    current_unix_time = int(time.time())
    return path_fixed.format(current_unix_time)


def load_latest_parquet_path() -> Path:
    """
    :return: path to latest parquet file storage
    """
    with open(SNAPSHOT.joinpath('pyspark', 'versions.yaml')) as f:
        content = yaml.safe_load(f)
        print(content)
        latest = content['latest']
        path = content[latest]['path']
        return SNAPSHOT.joinpath(path)



def show_review_text_stat(df: pd.DataFrame) -> None:
    """
    Show the distribution of review text length
    :param df: A dataframe
    :return: Nothing
    """
    stat = df['review_text_len'].describe().to_dict() 
    weird_reviews = len(df[df['review_text_len'] <= 1])
    print("Review Length Stat")
    pprint(stat)
    print(f"Reviews with length one or less: {weird_reviews}")

show_review_text_stat(raw_pdf)

print('Code Executed Successfully')

React

Dart

GoJS React

Typescript React

Vue.js

After successful code execution, we’ll see the message “Code Executed Successfully” in the terminal.

First, we create a new column by calculating the length of the text of the review text column. Then we add a function that takes a DataFrame. We use the `describe` method on the calculated column to show the `quartile`, `min`, `max` and `mean` values of the column. When creating the new column, we have to convert the datatype of the column first to make sure the `len` method will work.


Learn how to calculate statistics in pandas and PySpark.


Distribution of the Review Text Length

Learn how to calculate statistics in pandas and PySpark.

Introduction

Data Input/Output

Data Transformation

User Defined Function (UDF)

Wrapping Up

Appendix

Distribution of the Review Text Length

Calculate the statistics of the review text length in pandas