What is the token module in Python?
Overview
The token module in Python comprises tokens that represent the numeric values of leaf nodes of the parse tree.
The different methods in the module are as follows:
The tok_name constant
The tok_name constant returns a dictionary that maps the numeric values of the constants specified in this module back to name strings, enabling the generation of more human-readable representations of parse trees.
token.tok_name
The ISTERMINAL method
The ISTERMINAL method returns True if the given token is a terminal token. Otherwise, it returns False.
token.ISTERMINAL(x)
Note: The
xrefers to the given token.
The ISNONTERMINAL method
The ISNONTERMINAL method returns True if the given token is not a terminal token. Otherwise, it returns False.
token.ISNONTERMINAL(x)
Note: The
xrefers to the given token.
ISEOF method
The ISEOF method returns True if the given token is a marker token indicating the end of input.
token.ISEOF(x)
Note:
xrefers to the given token.
Example
import tokentk = token.PLUSprint("token.ISTERMINAL(%s) = %s" % (tk, token.ISTERMINAL(tk)))tk = token.COLONprint("token.ISNONTERMINAL(%s) = %s" % (tk, token.ISNONTERMINAL(tk)))tk = token.NEWLINEprint("token.ISEOF(%s) = %s" % (tk, token.ISEOF(tk)))print("token.tok_name = %s" % (token.tok_name,))
Explanation
- Line 1: We import the
tokenmodule. - Lines 3–13: We call the different methods outlined in the above sections with different inputs and print the output from the respective methods.