What is the token module in Python?

Overview

The token module in Python comprises tokens that represent the numeric values of leaf nodes of the parse tree.

The different methods in the module are as follows:

The tok_name constant

The tok_name constant returns a dictionary that maps the numeric values of the constants specified in this module back to name strings, enabling the generation of more human-readable representations of parse trees.

token.tok_name

The ISTERMINAL method

The ISTERMINAL method returns True if the given token is a terminal token. Otherwise, it returns False.

token.ISTERMINAL(x)

Note: The x refers to the given token.

The ISNONTERMINAL method

The ISNONTERMINAL method returns True if the given token is not a terminal token. Otherwise, it returns False.

token.ISNONTERMINAL(x)

Note: The x refers to the given token.

ISEOF method

The ISEOF method returns True if the given token is a marker token indicating the end of input.

token.ISEOF(x)

Note: x refers to the given token.

Example

import token
tk = token.PLUS
print("token.ISTERMINAL(%s) = %s" % (tk, token.ISTERMINAL(tk)))
tk = token.COLON
print("token.ISNONTERMINAL(%s) = %s" % (tk, token.ISNONTERMINAL(tk)))
tk = token.NEWLINE
print("token.ISEOF(%s) = %s" % (tk, token.ISEOF(tk)))
print("token.tok_name = %s" % (token.tok_name,))

Explanation

  • Line 1: We import the token module.
  • Lines 3–13: We call the different methods outlined in the above sections with different inputs and print the output from the respective methods.

Free Resources