Tokens in C
A sentence in English is made of words and punctuation marks. A C program is made of small pieces too. These pieces are called tokens.
In this lesson you will learn the 6 types of tokens and how to count tokens in a statement, a question that comes in almost every C exam.
What You Will Learn
- What a token is.
- The 6 types of tokens in C, with examples.
- How to break a statement into tokens.
- How to count tokens correctly.
Tokens in C Explained
Exam definition: A token is the smallest individual unit of a C program that has meaning to the compiler. C has six types of tokens: keywords, identifiers, constants, strings, operators and special symbols.
| # | Token type | Meaning | Examples |
|---|---|---|---|
| 1 | Keywords | Reserved words with a fixed meaning | int, if, return |
| 2 | Identifiers | Names given by the programmer | sum, main, total_marks |
| 3 | Constants | Fixed values that do not change | 10, 3.14, 'A' |
| 4 | Strings | Text inside double quotes | "Hello", "Sum = %d" |
| 5 | Operators | Symbols that do an operation | +, -, =, ==, && |
| 6 | Special symbols | Punctuation with a special job | ; , ( ) { } [ ] # |
Memory trick: "KICS OS" = Keywords, Identifiers, Constants, Strings, Operators, Special symbols.
Rules for counting tokens
- Each keyword, identifier and constant = 1 token.
- A whole string like
"Sum = %d"= 1 token, even with spaces inside. - Each operator = 1 token. A two-character operator like
==,++or<=is still 1 token. - Each special symbol (
;,,,(,)) = 1 token. Brackets count separately:(and)are 2 tokens. - Spaces and comments are not tokens.
Example 1: Count the Tokens in a Statement
int a = b + 10;
| # | Token | Type |
|---|---|---|
| 1 | int | Keyword |
| 2 | a | Identifier |
| 3 | = | Operator |
| 4 | b | Identifier |
| 5 | + | Operator |
| 6 | 10 | Constant |
| 7 | ; | Special symbol |
Output (answer)
Total tokens = 7
Explanation: How to Count Step by Step
- Read from left to right.
- Every time the type of character changes (word → symbol → number), a new token starts.
- Ignore spaces.
a=b+10;anda = b + 10 ;have the same tokens. - Write each token in a list and count the list.
How the Compiler Uses Tokens
- The first job of the compiler is called lexical analysis. The part that does it is called the lexer (or scanner).
- The lexer reads your code character by character and groups the characters into tokens.
- It always takes the longest possible token. That is why
==is one token, not two=tokens. - Then the compiler checks if the tokens are in a valid order (the grammar of C).
Example 2: Tokens in a Complete Program
#include <stdio.h>
int main(void)
{
int sum = 25 + 15; /* keywords, identifiers, constants */
printf("Sum = %d\n", sum); /* a string token and symbols */
return 0;
}
Output
Sum = 40
Count the tokens in printf("Sum = %d\n", sum);:
printf | ( | "Sum = %d\n" | , | sum | ) | ; = 7 tokens. The whole string is just one token.
Important Points to Remember
- A token is the smallest unit of a C program.
- There are 6 types of tokens.
- A string in double quotes is one token.
==,++,<=are each one token.- Spaces and comments are not tokens.
Common Mistakes While Counting Tokens
| Mistake | Correct way |
|---|---|
| Counting every word inside a string | The whole string "Hello World" = 1 token |
Counting == as 2 tokens | == is 1 operator token |
Counting () as 1 token | ( and ) are 2 tokens |
| Counting spaces | Spaces are not tokens |
Forgetting the ; at the end | The semicolon is a token too |
Tokens in C Practice Questions
- What is a token? Name the 6 types.
- Count the tokens:
float area = 3.14 * r * r; - Count the tokens:
if (x >= 10) - Is
"I love C"one token or three? - Which part of the compiler breaks code into tokens?
Show answers
- The smallest meaningful unit of a C program. Types: keywords, identifiers, constants, strings, operators, special symbols.
- 9 tokens:
float,area,=,3.14,*,r,*,r,; - 6 tokens:
if,(,x,>=,10,) - One token. A string in double quotes is a single token.
- The lexer (lexical analyser), in the lexical analysis phase.
Coding Exercise
Write a program that prints the result of count = count + 1; starting with count = 5. Then count the tokens in the line count = count + 1; and print that number too.
Show solution
#include <stdio.h>
int main(void)
{
int count = 5;
count = count + 1;
printf("count = %d\n", count);
printf("Tokens in 'count = count + 1;' = %d\n", 6);
return 0;
}
count = 6 Tokens in 'count = count + 1;' = 6
The 6 tokens are: count, =, count, +, 1, ;
Quick Quiz: Tokens in C (MCQ)
-
How many types of tokens are there in C?
A) 4 B) 5 C) 6 D) 8
-
How many tokens are in
x = y * 2;?A) 4 B) 5 C) 6 D) 7
-
"Hello World"is how many tokens?A) 1 B) 2 C) 3 D) 11
-
;is which type of token?A) Operator B) Constant C) Special symbol D) Keyword
-
Which is NOT a token?
A)
whileB)==C) A space D)100
Show answers
- C: 6.
- C: 6 tokens (
x,=,y,*,2,;). - A: 1 token.
- C: special symbol.
- C: spaces are not tokens.
Quick Revision for Exams and Interviews
Read this before your exam. The highlighted words are the answer.
Must-remember facts
| Token | Smallest meaningful unit of a program |
|---|---|
| Types (6) | Keywords, Identifiers, Constants, Strings, Operators, Special symbols |
| String token | Whole "..." = 1 token |
| Two-char operators | ==, ++, <= = 1 token each |
| Not tokens | Spaces and comments |
int a = b + 10; | 7 tokens |
| Made by | Lexer (lexical analysis) |
Frequently asked questions
- What is a token in C?
- The smallest individual unit of a program that the compiler understands.
- Is a string one token or many?
- One token, even if it has spaces inside.
- Is == one token or two?
- One token. The lexer takes the longest possible token.
- Are comments counted as tokens?
- No. Comments are removed before tokens are made.
Remember
- 6 types: KICS OS.
- Count rule: strings = 1,
=== 1, spaces = 0.
Next Topic
Next lesson: Variables in C. Module 02 starts with variables: how to store data in your program.