Search⌘ K
Home
Courses
Text Preprocessing with Python

Solution Explanations: Indexing

Explore various indexing methods used in text preprocessing, including term-based, document-based, and inverted indexing. Understand how these approaches organize and retrieve textual data efficiently. By the end of the lesson, you'll be able to implement and explain these indexing solutions using Python for improved natural language processing workflows.

We'll cover the following...

Solution 1: Term-based indexing

Here’s the solution:

Python 3.8
import pandas as pd
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from collections import defaultdict 
 
feedback_df = pd.read_csv("feedback.csv") 
feedback_df['tokens'] = feedback_df['feedback'].apply(lambda text: word_tokenize(text.lower()))
stop_words = set(stopwords.words('english'))
feedback_df['tokens'] = feedback_df['tokens'].apply(lambda tokens: [token for token in tokens if token not in stop_words])
index = defaultdict(list)
for idx, tokens in feedback_df[['feedback_id', 'tokens']].itertuples(index=False):
    for term in tokens:
        index[term].append(idx) 
for term in index.items():
    print(f"Term: {term}")

Let’s go through the solution explanation:

  • Line 8: We apply a lambda function to tokenize each feedback text and then convert it to lowercase using word_tokenize.

  • Lines 9–10: We initialize a set named stop_words with common English stopwords from the stopwords.words('english') list and further process the tokens column by applying another lambda ...