Skip to content
StudyFame

Tokens in C

A sentence in English is made of words and punctuation marks. A C program is made of small pieces too. These pieces are called tokens.

In this lesson you will learn the 6 types of tokens and how to count tokens in a statement, a question that comes in almost every C exam.

What You Will Learn

  • What a token is.
  • The 6 types of tokens in C, with examples.
  • How to break a statement into tokens.
  • How to count tokens correctly.

Tokens in C Explained

Exam definition: A token is the smallest individual unit of a C program that has meaning to the compiler. C has six types of tokens: keywords, identifiers, constants, strings, operators and special symbols.

#Token typeMeaningExamples
1KeywordsReserved words with a fixed meaningint, if, return
2IdentifiersNames given by the programmersum, main, total_marks
3ConstantsFixed values that do not change10, 3.14, 'A'
4StringsText inside double quotes"Hello", "Sum = %d"
5OperatorsSymbols that do an operation+, -, =, ==, &&
6Special symbolsPunctuation with a special job; , ( ) { } [ ] #

Memory trick: "KICS OS" = Keywords, Identifiers, Constants, Strings, Operators, Special symbols.

Rules for counting tokens

  • Each keyword, identifier and constant = 1 token.
  • A whole string like "Sum = %d" = 1 token, even with spaces inside.
  • Each operator = 1 token. A two-character operator like ==, ++ or <= is still 1 token.
  • Each special symbol (;, ,, (, )) = 1 token. Brackets count separately: ( and ) are 2 tokens.
  • Spaces and comments are not tokens.

Example 1: Count the Tokens in a Statement

int a = b + 10;
#TokenType
1intKeyword
2aIdentifier
3=Operator
4bIdentifier
5+Operator
610Constant
7;Special symbol

Output (answer)

Total tokens = 7

Explanation: How to Count Step by Step

  1. Read from left to right.
  2. Every time the type of character changes (word → symbol → number), a new token starts.
  3. Ignore spaces. a=b+10; and a = b + 10 ; have the same tokens.
  4. Write each token in a list and count the list.

How the Compiler Uses Tokens

  • The first job of the compiler is called lexical analysis. The part that does it is called the lexer (or scanner).
  • The lexer reads your code character by character and groups the characters into tokens.
  • It always takes the longest possible token. That is why == is one token, not two = tokens.
  • Then the compiler checks if the tokens are in a valid order (the grammar of C).

Example 2: Tokens in a Complete Program

#include <stdio.h>

int main(void)
{
    int sum = 25 + 15;            /* keywords, identifiers, constants */
    printf("Sum = %d\n", sum);    /* a string token and symbols */
    return 0;
}

Output

Sum = 40

Count the tokens in printf("Sum = %d\n", sum);:

printf  |  (  |  "Sum = %d\n"  |  ,  |  sum  |  )  |  ;  = 7 tokens. The whole string is just one token.

Important Points to Remember

  • A token is the smallest unit of a C program.
  • There are 6 types of tokens.
  • A string in double quotes is one token.
  • ==, ++, <= are each one token.
  • Spaces and comments are not tokens.

Common Mistakes While Counting Tokens

MistakeCorrect way
Counting every word inside a stringThe whole string "Hello World" = 1 token
Counting == as 2 tokens== is 1 operator token
Counting () as 1 token( and ) are 2 tokens
Counting spacesSpaces are not tokens
Forgetting the ; at the endThe semicolon is a token too

Tokens in C Practice Questions

  1. What is a token? Name the 6 types.
  2. Count the tokens: float area = 3.14 * r * r;
  3. Count the tokens: if (x >= 10)
  4. Is "I love C" one token or three?
  5. Which part of the compiler breaks code into tokens?
Show answers
  1. The smallest meaningful unit of a C program. Types: keywords, identifiers, constants, strings, operators, special symbols.
  2. 9 tokens: float, area, =, 3.14, *, r, *, r, ;
  3. 6 tokens: if, (, x, >=, 10, )
  4. One token. A string in double quotes is a single token.
  5. The lexer (lexical analyser), in the lexical analysis phase.

Coding Exercise

Write a program that prints the result of count = count + 1; starting with count = 5. Then count the tokens in the line count = count + 1; and print that number too.

Show solution
#include <stdio.h>

int main(void)
{
    int count = 5;
    count = count + 1;

    printf("count = %d\n", count);
    printf("Tokens in 'count = count + 1;' = %d\n", 6);
    return 0;
}
count = 6
Tokens in 'count = count + 1;' = 6

The 6 tokens are: count, =, count, +, 1, ;

Quick Quiz: Tokens in C (MCQ)

  1. How many types of tokens are there in C?

    A) 4   B) 5   C) 6   D) 8

  2. How many tokens are in x = y * 2;?

    A) 4   B) 5   C) 6   D) 7

  3. "Hello World" is how many tokens?

    A) 1   B) 2   C) 3   D) 11

  4. ; is which type of token?

    A) Operator   B) Constant   C) Special symbol   D) Keyword

  5. Which is NOT a token?

    A) while   B) ==   C) A space   D) 100

Show answers
  1. C: 6.
  2. C: 6 tokens (x, =, y, *, 2, ;).
  3. A: 1 token.
  4. C: special symbol.
  5. C: spaces are not tokens.

Quick Revision for Exams and Interviews

Read this before your exam. The highlighted words are the answer.

Must-remember facts

TokenSmallest meaningful unit of a program
Types (6)Keywords, Identifiers, Constants, Strings, Operators, Special symbols
String tokenWhole "..." = 1 token
Two-char operators==, ++, <= = 1 token each
Not tokensSpaces and comments
int a = b + 10;7 tokens
Made byLexer (lexical analysis)

Frequently asked questions

What is a token in C?
The smallest individual unit of a program that the compiler understands.
Is a string one token or many?
One token, even if it has spaces inside.
Is == one token or two?
One token. The lexer takes the longest possible token.
Are comments counted as tokens?
No. Comments are removed before tokens are made.

Remember

  • 6 types: KICS OS.
  • Count rule: strings = 1, == = 1, spaces = 0.

Next Topic

Next lesson: Variables in C. Module 02 starts with variables: how to store data in your program.