Text Preprocessing with Python/

...

/

Solution Explanations: Indexing

Solution Explanations: Indexing

Review solution explanations for the code challenges on indexing.

We'll cover the following...

Solution 1: Term-based indexing

Here’s the solution:

Python 3.8
import pandas as pd
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from collections import defaultdict 
 
feedback_df = pd.read_csv("feedback.csv") 
feedback_df['tokens'] = feedback_df['feedback'].apply(lambda text: word_tokenize(text.lower()))
stop_words = set(stopwords.words('english'))
feedback_df['tokens'] = feedback_df['tokens'].apply(lambda tokens: [token for token in tokens if token not in stop_words])
index = defaultdict(list)
for idx, tokens in feedback_df[['feedback_id', 'tokens']].itertuples(index=False):
    for term in tokens:
        index[term].append(idx) 
for term in index.items():
    print(f"Term: {term}")

Let’s go through the solution explanation:

  • Line 8: We apply a lambda function to tokenize each feedback text and then convert it to lowercase using word_tokenize.

  • Lines 9–10: We initialize a set named stop_words with common English stopwords from the stopwords.words('english') list and further process the tokens column by applying another lambda ...