Tokenization

From Text to Tokens — Understanding BPE and Vocabulary Size

When we work with Large Language Models (LLMs), the model does not directly process text the same way humans do.

Before text is given to the model, it goes through an important process called tokenization.

In this article, we will understand what tokenization is, how BPE works, what vocabulary size means, how to choose a vocabulary size, and how to evaluate a domain-specific tokenizer.

What Is Tokenization?

Tokenization is the process of splitting text into smaller pieces called tokens.

For example: I love Machine Learning

can be split into: [“I”, “love”, “Machine”, “Learning”]

Each of these pieces is called a token.

There are three common approaches to tokenization:

• Word-Level Tokenization
• Character-Level Tokenization
• Subword-Level Tokenization

1. Word-Level Tokenization

In word-level tokenization, each word becomes a token.

For example: low lower lowest

becomes: [“low”, “lower”, “lowest”]

The main problem with word-level tokenization is that the vocabulary can become very large.

It can also have difficulty handling words that were not present in the vocabulary.

2. Character-Level Tokenization

In character-level tokenization, text is split into individual characters.

For example: I love Machine Learning

can become: [“I”, “l”, “o”, “v”, “e”, “M”, “a”, “c”, “h”, “i”, “n”, “e”, …]

The disadvantage is that words are broken into very small pieces.

As a result, the number of tokens can become very large.

3. Subword-Level Tokenization

Subword tokenization takes a middle approach.

Instead of treating the entire word as one token or breaking it completely into individual characters, a word can be represented using smaller pieces.

For example: Learning

can be represented as: [“Learn”, “ing”]

Common approaches used for subword tokenization include BPE, WordPiece, and SentencePiece.

Understanding BPE — Byte Pair Encoding

One of the important concepts in subword tokenization is BPE, or Byte Pair Encoding.

The basic idea behind BPE is simple: Start with smaller pieces and repeatedly merge frequently occurring adjacent pairs.

Let’s understand this with a simple example. Suppose our training data contains:

low lower

Initially, we can represent them as individual characters:

l,o,w, l,o,w,e,r

Step 1: Find Frequent Adjacent Pairs

Now look at the adjacent pairs:

l o
o w

l o
o w
w e
e r

We can see that “l o” and “o w” occur more than once.

BPE looks at these frequencies and learns useful merges.

Step 2: Merge a Frequent Pair

Suppose BPE chooses:

l + o → lo

Now our words become:

low → lo,w

lower → lo,w,e,r

Step 3: Look for Frequent Pairs Again

Now we can see that:

lo + w

appears in both words.

So BPE can merge:

lo + w → low

Now: low → low, lower → low,e,r

The important point is that BPE repeatedly looks at adjacent pieces and learns merges based on their frequency.

It does not simply merge repeated words. It specifically learns frequent adjacent token pairs. As more merges are performed, larger token pieces can be created.

The final vocabulary depends on how many tokens we decide to keep, along with the initial token set and the learned merges.

What Is Vocabulary Size?

Now we come to an important question: How many tokens should our tokenizer have?

This is called the vocabulary size.

For example:

Vocabulary size = 16K means that the tokenizer has approximately 16,000 vocabulary entries.

Similarly:

32K → approximately 32,000 entries

64K → approximately 64,000 entries

Vocabulary size is important because it affects how text is represented.

Byte-Level BPE:

Traditional BPE can start from characters. Byte-Level BPE instead starts from UTF-8 bytes.

Unicode tells us what a character is, while UTF-8 tells us how that character is encoded into bytes.

A character can take 1–4 bytes in UTF-8, and every byte has a value from 0 to 255. Therefore, a byte-level tokenizer starts with a fixed base vocabulary of 256 possible byte values.

The BPE algorithm itself does not change:

Raw text
→ UTF-8 bytes
→ count adjacent byte/token pairs
→ merge frequent pairs
→ repeat until the target vocabulary size is reached.

So the important difference is the starting unit:

Character BPE:
Character → BPE merges

Byte-Level BPE:
UTF-8 bytes → BPE merges

This makes the tokenizer robust to arbitrary Unicode text because every input can ultimately be represented as bytes.

GPT-1 used character-level BPE, while GPT-2 and later GPT-style tokenizers use byte-level approaches.

BPE vs WordPiece vs SentencePiece

BPE and WordPiece have a similar overall idea: start with smaller pieces and combine them into larger pieces.

The important difference is how they decide which pair to combine.

BPE chooses the pair with the highest frequency:

score(A,B) = frequency(AB)

WordPiece instead considers how strongly the two pieces occur together compared with their individual frequencies:

score(A,B) = frequency(AB) / (frequency(A) × frequency(B))

So BPE asks: “Which pair occurs most often?”

WordPiece asks: “Which pair occurs together more strongly than we would expect from their individual frequencies?”

WordPiece is commonly associated with models such as BERT. It also commonly uses `##` to indicate that a piece continues a word.

For example: playing → play + ##ing

SentencePiece is different again: it is primarily a tokenizer library/framework rather than one single tokenization algorithm. It can implement algorithms such as BPE and Unigram.

SentencePiece also makes whitespace visible using a marker such as `▁`.

For example: “I love NLP”

can conceptually be represented as: ▁I ▁love ▁NLP

What Happens When We Increase Vocabulary Size?

A larger vocabulary allows the tokenizer to learn more merges.

More merges can create longer tokens. Longer tokens can represent the same text using fewer tokens.

The general idea is:

Larger Vocabulary
↓
More possible merges
↓
Longer token pieces
↓
Fewer tokens per document

But there is a trade-off.

A larger vocabulary also increases the vocabulary-related cost for the model.

So the goal is not simply: “Make the vocabulary as large as possible.”

We need to find a reasonable balance.

Average Tokens per Document

One useful measurement is the average number of tokens produced for each document.

A document is simply one piece of text in the dataset.

For example, suppose we have three documents:

Document 1 → 100 tokens

Document 2 → 150 tokens

Document 3 → 200 tokens

The average is:

(100 + 150 + 200) / 3 = 150

So the average is:

150 tokens per document.

Generally, a lower average number of tokens per document means better token efficiency because the same amount of text is represented using fewer tokens.

However, a lower token count alone does not automatically mean that a tokenizer is better.

We also need to consider vocabulary size, domain performance, and other evaluation measures.

Comparing Different Vocabulary Sizes

In the class example, different vocabulary sizes were tested.

The approximate average tokens per document were:

16K vocabulary → 188.8 tokens/document

32K vocabulary → 177.2 tokens/document

50K vocabulary → 173.1 tokens/document

64K vocabulary → 171.5 tokens/document

100K vocabulary → 169.7 tokens/document

We can see a clear pattern.

As vocabulary size increases, the average number of tokens per document decreases.

Comparing 16K and 32K Vocabulary

Let’s compare 16K and 32K.

16K → 188.8 tokens/document

32K → 177.2 tokens/document

The difference is:

188.8 − 177.2 = 11.6 tokens

So, the 32K vocabulary uses approximately 11.6 fewer tokens per document.

This is about a 6.2% reduction.

If we have 1,000 documents:

11.6 × 1,000 = 11,600

That means approximately 11,600 fewer tokens across 1,000 documents.

Get Sreekanth’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This demonstrates why vocabulary size matters.

The Trade-Off:

At first glance, we might think:

“If a bigger vocabulary gives fewer tokens, why not always use a huge vocabulary?”

Because there is a trade-off.

Increasing the vocabulary:

• Allows more merges
• Can create longer token pieces
• Can reduce the number of tokens per document
• But also increases vocabulary-related cost

Therefore, we need to find a point where increasing the vocabulary further provides only a small improvement.

The Knee Rule

This leads us to the Knee Rule.

The Knee Rule looks at how the average tokens per document changes as vocabulary size increases.

Initially, increasing vocabulary size can produce a noticeable reduction in the number of tokens.

But after a certain point, increasing the vocabulary further gives only a small improvement.

This point is called the “knee.”

For example:

16K → 188.8 tokens/document

32K → 177.2 tokens/document

50K → 173.1 tokens/document

64K → 171.5 tokens/document

100K → 169.7 tokens/document

The improvement from 16K to 32K is much larger than the improvement we get from increasing the vocabulary further.

So we look for the point where the improvement starts becoming smaller.

That point can be a reasonable vocabulary size to consider.

In the class example, 32K acts as a useful middle ground.

This does not mean that 32K is the perfect vocabulary size for every dataset.

The Corpus Rule

Another idea discussed in the class is the Corpus Rule.

BPE learns its vocabulary by performing merges.

Therefore, we need enough training data for those merges to be meaningful and stable.

The class discusses a 10× rule of thumb as a bare minimum.

For example:

Vocabulary Size = 16K

16K × 10 = 160K

The basic idea is that we need enough training data relative to the vocabulary size.

The class also discusses that 20× or 30× can be used.

The exact amount depends on the dataset and situation.

What If the Knee Rule and Corpus Rule Disagree?

The Knee Rule and Corpus Rule answer different questions.

The Knee Rule asks:“At what vocabulary size do additional vocabulary slots stop giving meaningful token-efficiency improvements?”

The Corpus Rule asks: “Do I have enough training data to support this many vocabulary entries?”

Therefore, they can disagree.

For example:

Knee Rule: → 64K or 100K looks efficient

Corpus Rule: → 16K or 32K is better supported by the available corpus size

In this situation, we should not automatically choose the largest vocabulary.

For the example in this course, 16K → 32K gives a meaningful 6.2% reduction in tokens per document, while further increases provide much smaller improvements.

Because the corpus is relatively small and a larger vocabulary increases model cost, 32K becomes the practical choice.

So the final vocabulary size is a trade-off between:

Token efficiency
+ Training data availability
+ Model cost
+ Domain requirements

Fertility

Another important concept when evaluating a tokenizer is fertility.

Fertility means:

The average number of tokens produced for each word.

For example, suppose: 3 words → 6 tokens

Then: Fertility = 6 / 3

Fertility = 2

So the fertility is: 2 tokens per word.

High Fertility vs Low Fertility

High fertility means that a word is being split into many tokens.

For example:

One word
↓
Many tokens

Low fertility means:

One word
↓
Fewer tokens

There is no single perfect fertility value for every dataset.

It depends on the dataset and the domain.

For a domain-specific tokenizer, lower fertility on the target domain is generally better.

Domain-Specific Tokenization

A general-purpose tokenizer is trained using general text.

But specialized domains can contain terminology that may not appear frequently in general text.

For example, medical text contains many specialized medical terms.

A domain-specific tokenizer is trained using data from a particular domain.

The main goal is to make the tokenizer more efficient for that domain.

How Do We Evaluate a Domain-Specific Tokenizer?

The class discusses several important measurements.

Domain Fertility

First, we measure fertility on the target domain.

For example, if we are building a medical tokenizer, we test it on medical data.

The goal is:

Lower fertility on the target domain.

This means domain words are generally being represented using fewer tokens.

Single-Token Ratio

Another useful measurement is the Single-Token Ratio.

The question is:

How many important domain terms are represented as a single token?

For example, suppose we test 100 important domain terms.

If 70 terms are represented by exactly one token:

Single-Token Ratio = 70 / 100 = 70%

A higher single-token ratio means more important domain terms are represented as single tokens.

How Should We Evaluate a Domain-Specific Tokenizer?

The evaluation should happen on held-out data, not only on the data used to train the tokenizer.

The priority is:

1. Domain fertility — primary metric
2. Single-token rate — secondary metric
3. General-purpose sanity check — final check

The goal is not to make the tokenizer perform well only on a handful of manually selected examples.

Instead, evaluate it on a representative held-out dataset from the target domain.

A tokenizer that performs extremely well on the training corpus but poorly on unseen domain text has not necessarily solved the problem.

General-Purpose Sanity Check

We should not test the tokenizer only on domain data.

We should also test it on general-purpose data.

The expected behavior is:

Domain-specific tokenizer
↓
Better performance on the target domain
↓
May perform worse on general-purpose data

This acts as a sanity check.

It helps us confirm that the tokenizer has actually become specialized for its intended domain.

Why Is Frequency Important in BPE?

Frequency is an important part of BPE.

BPE learns merges based on how frequently adjacent pairs occur.

For example, if a pair appears many times in the training data, there is stronger evidence that the pair is useful.

If a pair appears only once, there is much less evidence that it should become an important token.

Therefore, the amount of training data and the frequency of patterns both matter when learning a vocabulary.

19. Putting Everything Together

When building a tokenizer, we should not choose vocabulary size randomly.

We can think about the process like this:

Training Data

↓

Train Tokenizers with Different Vocabulary Sizes

↓

Measure Average Tokens per Document

↓

Look for the Knee

↓

Check Whether There Is Enough Training Data

↓

Evaluate Domain Fertility

↓

Check Single-Token Ratio

↓

Perform General-Purpose Sanity Check

↓

Choose a Reasonable Vocabulary Size

Final Takeaway

Tokenization is the process of converting text into smaller pieces called tokens.

1. Word-level tokenization treats words as tokens.
2. Character-level tokenization breaks text into individual characters.
3. Subword tokenization takes a middle approach.
4. BPE is a subword tokenization approach that repeatedly merges frequently occurring adjacent pairs.
5. Vocabulary size determines how many token entries the tokenizer can have.
6. A larger vocabulary can result in fewer tokens per document because it allows more merges and longer token pieces. However, a larger vocabulary also increases vocabulary-related cost. Therefore, we should not simply choose the largest vocabulary.Instead, we should look for a reasonable balance.
7. The Knee Rule helps us identify where increasing vocabulary size starts giving smaller improvements.
8. The Corpus Rule and 10× rule remind us that we need enough training data to support the vocabulary and its learned merges.

For a domain-specific tokenizer, we should also evaluate:

• Domain Fertility
• Single-Token Ratio
• General-Purpose Sanity Check
• Average Tokens per Document
• Training Data Size

The most important lesson is: There is no single perfect vocabulary size.

The right vocabulary size depends on the dataset, domain, amount of training data, token efficiency, and the trade-off between vocabulary size and cost.