Page 56 - Touhpad Ai

Tokens

A token is the smallest unit of a program that has a meaning. Tokens in Python are fundamental elements of the language’s
syntax. They represent the building blocks of Python code, consisting of keywords, identifiers, literals, operators, and
punctuations. Understanding tokens is essential for comprehending Python’s lexical structure (basic syntax and set of
rules defining how Python programs are written) and parsing rules (guidelines used by a parser to analyse the structure
of code).
Python code is first divided into tokens during the lexical analysis phase of the interpretation process. These tokens are
then used by the parser to construct the abstract syntax tree (AST a tree representation of Python code that is used for
code analysis and manipulation), which is further processed to execute the code.

Tokens

Keywords Identifiers Literals Operators Punctuators

Keywords
Keywords are predefined and reserved words in Python, which have special meanings and purposes. They cannot be
used as identifiers or variable names. These keywords are part of Python’s syntax and are used to define control flow,
declare functions and classes, handle exceptions, etc.

Following table shows the keywords used in Python:
and as assert async await
break class continue def del
elif else except False finally
for from global if import

in is lambda None nonlocal
not or pass raise return
True try while with yield

Identifiers
Identifiers are the names used to identify variables, functions, classes, modules, and other objects in Python. They act
as labels for these elements, allowing you to refer to them in your code. Identifiers follow certain rules and conventions:
Must begin with a letter (a-z, A-Z) or an underscore (_).
Subsequent characters can be letters, digits (0-9), or underscores.
Case-sensitive (myVar is different from myvar).
Cannot be a Python keyword (reserved words).
No special characters such as !, @, #, $, %, etc., are allowed within identifiers.
Blank spaces within an identifier are disallowed.
For example:

my_variable
calculate_sum
MyClass
Some conventions that can be used while specifying identifiers for more clarity are as follows:
Use descriptive names that convey the purpose of the variable, function, etc.

54 Touchpad Artificial Intelligence - XI