Trusted answers to developer questions
Trusted Answers to Developer Questions

Related Tags

python
communitycreator
token
module

What is the token module in Python?

abhilash

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Overview

The token module in Python comprises tokens that represent the numeric values of leaf nodes of the parse tree.

The different methods in the module are as follows:

The tok_name constant

The tok_name constant returns a dictionary that maps the numeric values of the constants specified in this module back to name strings, enabling the generation of more human-readable representations of parse trees.

token.tok_name

The ISTERMINAL method

The ISTERMINAL method returns True if the given token is a terminal token. Otherwise, it returns False.

token.ISTERMINAL(x)

Note: The x refers to the given token.

The ISNONTERMINAL method

The ISNONTERMINAL method returns True if the given token is not a terminal token. Otherwise, it returns False.

token.ISNONTERMINAL(x)

Note: The x refers to the given token.

ISEOF method

The ISEOF method returns True if the given token is a marker token indicating the end of input.

token.ISEOF(x)

Note: x refers to the given token.

Example

import token
tk = token.PLUS
print("token.ISTERMINAL(%s) = %s" % (tk, token.ISTERMINAL(tk)))
tk = token.COLON
print("token.ISNONTERMINAL(%s) = %s" % (tk, token.ISNONTERMINAL(tk)))
tk = token.NEWLINE
print("token.ISEOF(%s) = %s" % (tk, token.ISEOF(tk)))
print("token.tok_name = %s" % (token.tok_name,))

Explanation

  • Line 1: We import the token module.
  • Lines 3–13: We call the different methods outlined in the above sections with different inputs and print the output from the respective methods.

RELATED TAGS

python
communitycreator
token
module

Grokking Modern System Design Interview for Engineers & Managers

Ace your System Design Interview and take your career to the next level. Learn to handle the design of applications like Netflix, Quora, Facebook, Uber, and many more in a 45-min interview. Learn the RESHADED framework for architecting web-scale applications by determining requirements, constraints, and assumptions before diving into a step-by-step design process.

Keep Exploring