What is BPE? read more
Byte-Pair Encoding starts from a small base vocabulary — individual characters or raw UTF-8 bytes — and repeatedly merges the most frequent adjacent pair into a new token. Each merge grows the vocabulary by one. After many merges, common substrings (“the”, “ing”, whole words) become single tokens. This is how modern tokenizers (GPT-2, LLaMA, etc.) build their vocabularies.
Pre-tokenization splits text on spaces and punctuation first, so merges never cross word/punctuation boundaries (toggle it below to compare). Base tokens show their content (a character or byte); each merged token is shown as its new vocabulary ID (e.g. 256 (th)). Press Merge to step through the algorithm.
Adjacent token pairs, sorted by frequency. The top row is what Merge will combine.
| Rank | Pair | Count |
|---|
Every token the model knows, with its integer ID. Byte-level mode starts with all 256 bytes as base tokens; newly merged tokens are highlighted.
| ID | Token | Type |
|---|
Each merge in order. Click an entry to jump back to that step.
Apply Tokenize New Text
Type any text below. It is tokenized using the merges learned so far (applied in learned order). In character mode, characters never seen in training show up as red OOV chips; in byte mode every byte is always representable (the base vocab is all 256 bytes), so there is no OOV.