Assuming a vocabulary size of x, a token is just an integer value that satisfies:
0≤token<x
Remember NotaGen from the Symbolic Music Generation assignment? NotaGen generates abc notation, which is just text: Now we know one way that models can produce text
Byte Pair Encoding
Text-based LLMs typically use Byte Pair Encoding (BPE), which tokenizes groups of characters.
A robust BPE text tokenizer will help the model see common subword character sequences. In English "izer" is a common subword, so a BPE tokenizer might split "tokenizer" into tokens like token, and izer (instead of e.g. toke and nizer). This helps models generalize and "understand" grammar.
Once we have tokens, the transformer will use them to look up column vectors from an embedding table. Each token index corresponds to a column vector that represents that token numerically.
(space)
↓
0
↓
−1.0−0.9−1.0−1.4−0.9−0.6−0.0⋮−1.1
a
↓
1
↓
−1.0−1.3−1.7−1.6−0.5−1.6−1.7⋮−1.5
b
↓
2
↓
−1.0−1.7−0.4−1.7−1.8−1.4−0.6⋮−1.8
c
↓
3
↓
−1.0−2.0−0.8−1.9−0.9−0.4−1.2⋮−1.9
⋯
9
↓
67
↓
−0.9−1.2−1.5−0.7−0.5−0.2−0.5⋮−0.4
During the training process, the values in the embedding table are gradually adjusted. If we train successfully, these values will encode the semantic significance of the tokens in a high-dimensional space.
Resources
3Blue1Brown: But what is a GPT? Visual intro to transformers