The words generated by the linear analysis may be of different kinds:
identifier,
keyword (if, while, ...),
punctuation character,
multi-character operator (:=, ->, ...).
Such a kind is called a TOKEN and an element of a kind
is called a LEXEME.
A word is recognized to be a lexeme for a certain token by PATTERN MATCHING.
For instance letter followed by letters and digits is a pattern
that matches a word like x or y with the token id (= identifier).
Token
Lexeme
Pattern
ID
x y n0
letter followed by letters and digits
NUM
-123 1.456e-5
any numeric constant
IF
if
if
LPAREN
(
(
LITERAL
``Hello''
any string of characters (except ``) between `` and ``
