Tokens in Python

M3-R5.1 · Chapter 3: Introduction to Python · 4 min read

What is Token?

The smallest individual unit in a python program is known as Token or Lexical unit . All statements and instructions in a program are built with tokens. 

Each token should be separated from the other by a space,tab,carriage return which are collectively termed as White spaces.
Python has following tokens :

  1. Keywords – Reserved words with special
  2. Identifiers – Names for variables, functions, and classes
  3. Literals – Fixed values like 10, 3.14, 'Hello', True, None, etc.
  4. Punctuators – Special symbols used in syntax.
  5. Operators – Symbols for performing operations.

 1 ) Keywords 

  • keywords are reserved words that have special meanings and are used to define the syntax and structure of the language.
  • At the time of designing a language, some words are reserved to do specific tasks. Such words are called keywords or reserved words.
  • They cannot be used as Identifiers( variable names, function names, class name etc.)
  • NOTE : All the keywords are in lowercase except 03 keywords (True, False, None)  and is also called  constant keywords.

Keyword

Purpose

False

Represents the Boolean Value false.

True

Represents the Boolean Value true

None

Represents the absence of a value or a null value.

and

Logical AND operator

as

Used to create an alias (e.g. Import module as alias)

assert

Used for debugging and testing conditions

Async

Used to declare asynchronous function or coroutines

Await

Pauses execution until an async task is completed

Break

Exits a loop prematurely

Class

Defines a class

Continue

Skips the remaining loop iteration and moves to the next iteration

Def

Defines a function or method

Del

Deletes objects, variables or array elements

Elif

An alternative condition in conditional statements

Else

Specifies what to do when the condition in an if statements is false

Except

Specifies how to handle exceptions

Finally

Always runs after try-except blocks, used for cleanup code

For

Defines a loop that iterates over a sequence.

From

Specifies the module to import specific parts of

Global

Declares a global variable inside a function.

If

Starts a conditional statement

Import

Imports a module

In

Tests membership or iteration in a sequence

Is

Tests object identity

Lambda

Defines an anonymous function.

Nonlocal

Refers to a variable in the nearest enclosing scope that is not global

Not

Logical NOT operator

Or

Logical OR operator

Pass

Placeholder that does nothing

raise

Raises an exception

Return

Specifies the return value of a function

Try

Defines a block of code to test for errors

While

Defines a loop that runs as long as a condition is true

With

Simplifies exception handling and ensures proper cleanup

Yield

Pauses a function and returns a value , used in generators

  • We can print a list of python keywords through the below statements:

>>> help("keywords")

# show the list of keyword

>>> import keyword  OR   import keyword

>>>keyword.kwlist    OR print(keyword.kwlist)

# Make the module keyword available

# Syntax module name.object/function