How to create a dictionary using Gensim
Gensim is a powerful Python module widely used for natural language processing (NLP) tasks. One of its important features is the ability to generate dictionaries, which are essential in text analysis and modeling.
What is a Gensim dictionary?
A Gensim dictionary is a fundamental data structure that converts words into unique integer IDs. It acts as a lookup database for word-to-ID and ID-to-word conversions, allowing for fast textual data processing for different NLP applications.
Note: Make sure you have Gensim installed on your system. If not, you can install it using
pip install gensimcommand on your terminal.
Let's thoroughly understand the creation of a dictionary in a step-by-step approach.
Creating a dictionary with Gensim
To build a dictionary using this module, we start by importing the necessary libraries,
gensim,corpora, which provides functionality for working withcorpora Collections of text documents. pprintis a Python-based module and is imported for pretty printing.
import gensimfrom gensim import corporafrom pprint import pprint
Here, we define a list called
docthat contains sample documents. The documents are strings representing a text and will be used to create the dictionary.
doc = ["Machine learning is a subfield of artificial intelligence that focuses on the development of algorithms","and models that allow computers to learn and make predictions based on data."]
Now, we preprocess the text data before creating the Gensim dictionary. They are processes like tokenization, removing stop words, and converting the text into a suitable format for analysis.
We tokenize the documents by splitting each document into individual words. The resulting tokens are stored in the
text_tokenslist.
text_tokens = [[text for text in doc.split()] for doc in doc]
Finally, we create a Gensim dictionary using the
Dictionary()function from thecorporamodule. TheDictionary()function takes thetext_tokenslist as input and builds a dictionary that maps words to unique integer IDs.
dict_LoS = corpora.Dictionary(text_tokens)
We print the contents of the created dictionary at last.
Note: token2id It is an attribute of the Gensim dictionary object that maps words to their corresponding integer IDs.
print(dict_LoS.token2id)
By printing dict_LoS.token2id, we can see the dictionary in the format of {word: id} pairs.
The complete implementation of the Gensim dictionary can be seen below:
Code
import gensimfrom gensim import corporafrom pprint import pprint# Sample documentsdoc = ["Machine learning is a subfield of artificial intelligence that focuses on the development of algorithms","and models that allow computers to learn and make predictions based on data."]# Tokenize the documentstext_tokens = [[text for text in doc.split()] for doc in doc]# Create a dictionary from the tokenized documentsdict_LoS = corpora.Dictionary(text_tokens)# Print the dictionaryprint(dict_LoS.token2id)
By executing this code, you will have a Gensim dictionary (dict_LoS) that maps each word to a unique integer ID based on the provided sample documents.
Output
Upon execution, the code will tokenize the sample documents, create a Gensim dictionary based on the tokenized words, and then print the dictionary showing the terms and their respective integer IDs.
The output looks something like this:
Conclusion
In conclusion, by creating dictionaries using Gensim, developers can effectively convert text data into a structured format that facilitates efficient analysis and modeling. The Gensim dictionary allows for mapping words to unique integer IDs, facilitating text transformation into numerical representations that machine learning algorithms can process.
Free Resources