Tokens and its types in C++ with examples - Cyborg Coding

In C++, a token is the smallest unit of a program that can be understood by the compiler. Tokens are the basic building blocks of a C++ program, and they are grouped into different types. The main types of tokens in C++ are:

1. Keywords: These are predefined words in C++ that have a specific meaning and function in the language. Examples of keywords include "int", "float", "while", "if", and "else".

2. Identifiers: These are names given to variables, functions, and other objects in a C++ program. Identifiers must begin with a letter or an underscore and can include letters, digits, and underscores. For example, "x", "counter", and "calculate_average" are all valid identifiers.

3. Constants: These are fixed values that cannot be changed during the execution of a program. In C++, there are several types of constants, including integer constants, floating-point constants, character constants, and string literals. For example, "5", "3.14", 'a', and "hello" are all examples of constants.

4. Operators: These are special symbols that are used to perform operations on variables and constants. C++ supports several types of operators, including arithmetic operators (+, -, *, /), comparison operators (==, !=, >, <, >=, <=), and logical operators (&&, ||, !).

5. Punctuators: These are special symbols that are used to separate and organize the elements of a C++ program. Punctuators include the semicolon (;), the comma (,), and the curly braces ({}).

6. White spaces: These are spaces, tabs, and newline characters that are used to separate tokens in a C++ program. White spaces are ignored by the compiler, but they are important for making the code more readable.

Here are some examples of how these tokens are used in a C++ program:

#include <iostream>

using namespace std;

int main() 

{

    int x = 5;  // x is an identifier, "int" is a keyword, and "5" is a constant.

    int y = 7;  // y is an identifier, "int" is a keyword, and "7" is a constant.

    int z = x + y;  // + is an operator, x and y are identifiers, and z is an identifier.

    cout << z;  // "cout" is an identifier, "<<" is an operator, and "z" is an identifier.

    return 0;  // "return" is a keyword, "0" is a constant, and ";" is a punctuator.

}

In this example, we have several keywords ("include", "int", "using", "namespace", "std", "main", "cout", "return"), identifiers ("x", "y", "z", "cout"), constants ("5", "7", "0"), operators ("+", "<<"), and punctuators (";", "{", "}", "(", ")").

It is important to note that C++ is case-sensitive language, so "x" and "X" would be considered different identifiers. Also, C++ has a set of rules for naming conventions for variables, function, classes etc called C++ naming conventions.