Anatomy of an LLM: tokens and embeddings

🚀 Intro

In the previous stop of this series - one of the branches of the overview post “Anatomy of an LLM” - we assembled a whole network from neurons and showed how training turns random noise into a model that understands language. All along, though, we kept talking about “processing tokens” and “predicting the next token,” while passing over the most basic question: what actually is a token, and how does a word become a number that can be multiplied?

Because the network does not see letters. It does not see words. It sees vectors - sequences of numbers. Before any of the matrix multiplications from the previous post can even begin, the text has to go through two stages: first it is cut into pieces (tokens), and then each piece is given a vector (an embedding). This is the bridge between the world of characters and the world of numbers - and it has a surprisingly rich construction.

In this post we will see why modern models cut text neither into words nor into single characters, but into something in between. We will trace how that famous geometry of meaning is born, where directions in space correspond to concepts and subtracting vectors yields analogies. And - in the spirit of the whole series - we will look at it critically: because the example “king - man + woman ≈ queen” is a beautiful intuition, but one that is often over-interpreted. Finally we will touch on something primers usually skip: how the model knows in what order the words came at all.

💡 How to read this. The main thread is descriptive and self-contained. I tuck the formulas into expandable “for the curious” blocks - if you like math, open them; if not, skip them with no loss to understanding. Every claim has a footnote to its source paper at the end.

📋 TL;DR

• A token is not a word. Splitting into whole words blows up the vocabulary (especially in inflected languages like Polish) and trips over unknown words; splitting into characters makes sequences hopelessly long. The standard is subword tokenization - common words become a single token, rare ones split into known pieces.
• Subword algorithms: BPE (merges the most frequent pairs - the core of GPT), WordPiece (a probabilistic criterion - BERT), Unigram/SentencePiece (prunes a large vocabulary - language-independent).
• How many tokens? The vocabularies of today’s models range from about 50k to about 200k tokens. Rule of thumb: about 4 characters per token in English - much fewer in inflected languages, so the same text “costs” more tokens.
• An embedding is a vector of meaning. Each token gets a row in a trained embedding matrix $E$. The result is a space where closeness = similarity of meaning, and where directions can be interpretable.
• The famous analogy, with a critical eye. “King - man + woman ≈ queen” shows the linear structure of the space, but it does not prove that the model “understands” relations - it is partly an artifact of the metric and the choice of examples (Linzen; Drozd et al.).
• Static vs contextual. word2vec/GloVe give one vector per word (the same “bat” in “vampire bat” and “baseball bat”). Transformers give contextual representations - the vector depends on the sentence.
• Position. Attention by itself does not know the order of tokens - it has to be injected: sinusoidal encoding, learned tables, and today most often RoPE (rotation) and ALiBi (a linear penalty for distance).

👉 Play with it: on the interactive page you have a section on tokens and embeddings - watch how text breaks into pieces and how the vectors arrange themselves in the space of meaning.

✂️ Why neither words nor letters

It would seem most natural to split text into words. The trouble is that language is “open” - new words, proper names, blends and typos keep appearing. A vocabulary has to be finite, so anything not in it lands in a single “unknown” bin (the <UNK> token, from unknown). This is the so-called OOV problem (out-of-vocabulary).

In inflected languages - like Polish, Finnish or Turkish - it hurts twice over. A single word has dozens of forms (Polish kot, kota, kotu, kotem, kotów, kotami… - “cat” in various cases). If each form were a separate token, the vocabulary would explode, and the rarer forms would still be unknown to the model. Polish is practically a textbook adversary of the “word = token” approach.

The other extreme is splitting into single characters. OOV vanishes (the alphabet is small and closed), but a different problem appears: sequences become many times longer. A word like “running” is one or two subword tokens, but seven characters. And the cost of the attention mechanism grows with the square of the sequence length (we will get to that in the next post) - so longer sequences mean markedly more expensive computation. On top of that, the whole burden of composing characters into meaning then falls on the network itself.

Subword tokenization is the golden mean: common words stay as one token, while rare and unknown ones break into smaller, known pieces. The vocabulary has a controlled size, OOV practically disappears, and sequences stay short. This is today’s standard across all large language models.

🧩 Subword algorithms: BPE, WordPiece, Unigram

How does one actually decide which pieces should be tokens? This is where three families of algorithms come in.

Byte-Pair Encoding (BPE). Curiously, it was born as a data compression algorithm - Philip Gage described it in 1994¹. The idea is dead simple: find the most frequently adjacent pair of symbols and replace it with one new symbol; repeat. Sennrich, Haddow and Birch (2016) carried this idea over to language processing²: we start from single characters and in each step merge the most frequently co-occurring pair, until the vocabulary reaches a target size. The list of those merges becomes the recipe for tokenizing any new text.

GPT-2 added a clever variant: byte-level BPE, operating on UTF-8 bytes instead of characters³. Since every possible byte (there are 256) is in the base alphabet, any Unicode string can be encoded without <UNK> - zero OOV by construction. This approach reigns in the GPT family.

WordPiece. Google’s variant (Schuster and Nakajima, 2012)⁴, popularized by their translation system⁵ and used in BERT⁶. It differs from BPE in its merge criterion: instead of plain “most frequent pair,” WordPiece merges the pair that most increases the probability of the corpus under a simple statistical model. A probabilistic criterion, not a purely frequency-based one.

Unigram LM and SentencePiece. Kudo (2018) approached it from the other end⁷: instead of building the vocabulary bottom-up by merging, it starts from a large set of candidates and prunes it, removing the tokens whose absence hurts the least. SentencePiece (Kudo and Richardson, 2018) is a popular implementation that treats text as a raw stream of characters - it even encodes the space as an ordinary symbol (▁)⁸. This makes it language-independent, which is crucial for languages without spaces between words, like Japanese or Chinese.

In practice every large model has its own tokenizer with a specific vocabulary size. For the OpenAI family (the tiktoken library)⁹:

A practical rule of thumb from the tiktoken docs: one token is on average about 4 characters of English text¹⁰. Here is a catch that matters to us: most vocabularies are optimized mainly for English, so Polish text - with all its inflection and diacritics - breaks into more tokens than English of the same content. This is a real reason why the same content in Polish tends to be costlier and slower to process than in English. (This is a general observation, not a measured number from a specific paper.)

l o w ·        (×5)
l o w e r ·    (×2)
n e w e s t ·  (×6)
w i d e s t ·  (×3)

n e w es t ·   (×6)
w i d es t ·   (×3)

🗺️ Embeddings: from a number to meaning

We have tokens. But a token is still just a label - a row number in the vocabulary. The number alone means nothing: that “cat” has index 4711 and “dog” 4712 says nothing about them being close in meaning. We need to turn each token into a vector - a sequence of numbers carrying information about meaning. That is the embedding (literally “embedding” a word into a space of numbers).

The breakthrough was word2vec (Mikolov et al., 2013). The idea: learn word vectors so as to predict which words appear in the neighborhood. Two variants¹¹: CBOW (from the context, guess the missing middle word) and Skip-gram (from one word, guess its context; better for rare words). The team’s second paper added an efficient training method and - what fired imaginations - showed vector analogies¹². It turned out that in the learned space $\text{vec}(\text{"King"}) - \text{vec}(\text{"Man"}) + \text{vec}(\text{"Woman"})$ has as its nearest neighbor $\text{vec}(\text{"Queen"})$. The direction “maleness → femaleness” became an almost constant offset in the space. This is exactly that famous geometry of meaning that the interactive page shows.

But honesty is needed here - and this is the point where popular stories about LLMs often overreach. The classic example tends to be over-interpreted. First, the standard method of finding the answer excludes by design the input words themselves (“king,” “man,” “woman”) - without that exclusion, the nearest neighbor would often just be one of them. Linzen (2016) showed that simple baselines can be surprisingly strong, and that the method confuses “offset consistency” with the plain structure of the neighborhood¹³. Drozd, Gladkova and Matsuoka (2016) added that the linear offset is sensitive to the idiosyncrasies of individual words and works unevenly across different types of relations¹⁴.

The conclusion worth remembering: the analogy “king - man + woman ≈ queen” is a good intuition that the space has a linear structure - but not a proof that embeddings “understand” relations. It is partly an artifact of the metric and of carefully chosen examples. The interactive page shows this example uncritically (because it is an excellent visualization of the intuition) - here it is worth adding this caveat in the margin.

GloVe (Pennington, Socher, Manning, 2014) took a different road¹⁵. Instead of sliding a local window over the text, GloVe learns directly from global statistics: how often word $i$ co-occurs with word $j$ across the entire corpus. The effect is similar - a space with interpretable geometry - but the route is different.

Static vs contextual - and what is really in an LLM

word2vec and GloVe have one fundamental limitation: they are static. One word = one vector, regardless of the sentence. The word “bat” gets the same vector in “vampire bat” and “baseball bat” - even though these are two different things. The breakthrough came with contextual representations: ELMo¹⁶ and then BERT⁶ generate a word’s vector that depends on the whole sentence. “Bat” gets a different vector depending on its neighbors. This very ability is one of the foundations of today’s LLMs.

And here is a key correction, because it is an easy point to misunderstand: an LLM does not load ready-made word2vec vectors. The embedding layer is simply a matrix of parameters, trained together with the rest of the model - exactly like the weights from the previous post. Each of the $V$ tokens in the vocabulary has its row in the matrix $E$, with as many columns as the model dimension $d_{\text{model}}$ (read “d-model”). These vectors start out random and are polished by gradients during training - the network invents for itself whatever geometry of meaning is most convenient for predicting the next token.

How large are these vectors in real models? The base Transformer from 2017 had $d_{\text{model}} = 512$¹⁷. GPT-3 (175 billion parameters) - already $d_{\text{model}} = 12288$¹⁸. LLaMA scaled from 4096 (the 7B variant) to 8192 (the 65B variant)¹⁹. Every token is thus represented by thousands of numbers.

🧭 How the model knows in what order the words came

There is one more problem that primers usually pass over, and it is absolutely crucial. The attention mechanism - the heart of the Transformer, which we reach in the next post - has a surprising property: it is order-insensitive. By itself it does not tell “the dog bit the postman” apart from “the postman bit the dog.” It looks at a set of tokens, not at their arrangement. And yet word order carries meaning.

So information about position has to be injected separately. Several ideas emerged for how to do this.

Sinusoidal encoding (absolute). The original Transformer added to each embedding a fixed vector built from sines and cosines of different frequencies¹⁷ - something like a unique “fingerprint” of the position. Each position in the sentence gets its characteristic pattern, and the model learns to read it.

Learned encoding. A simpler alternative: a separate trained table of vectors, one per position (this is what BERT and GPT-2 do). The quality comes out almost identical to the sinusoids¹⁷, but there is a drawback: the model does not naturally handle sequences longer than those it saw in training - for a position it never trained on, it simply has no vector.

RoPE (Rotary Position Embedding). Today’s favorite (Su et al., 2021)²¹, used among others in LLaMA. Instead of adding position, RoPE rotates the query and key vectors by an angle proportional to position. The idea is elegant: after the rotation, the dot product of two vectors depends only on the difference of their positions, not on the absolute positions. The model thus encodes position relatively (“how much further along”), which better reflects how language actually works.

ALiBi (Attention with Linear Biases). Another clever idea (Press, Smith, Lewis, 2021)²²: do not touch the embeddings at all, just add to the attention scores a linear penalty that grows with the distance between tokens. The farther apart, the stronger the damping. The main advantage: a model trained on short sequences can extrapolate to much longer ones at inference time.

🌗 A geometry no one designed

Let us pause for a moment on what we have just described - because in the technical density it is easy to miss something astonishing. We started from discrete symbols: a token is just a number in a vocabulary, a label with no content. And we ended at a continuous space in which directions correspond to concepts, closeness means similarity, and relations (at least roughly) fall into geometric offsets. No one designed this geometry by hand. It emerged on its own - as a side effect of the only pressure we know from the previous post: “predict the next token as well as you can.”

That is why I kept the honest critique of the famous vector analogies in this post - not to strip the geometry of its wonder, but so as not to ascribe to it more than the data show: “king - man + woman ≈ queen” holds less often and less cleanly than the legend claims. And even so, something striking remains without any exaggeration - that discrete symbols, labels with no content, can be embedded at all in a space where distance begins to resemble similarity, and direction - relation.

And here comes the boldest hypothesis of the whole series, and at the same time the one that fits this post best - because it bears directly on the geometry of representation. It claims that the space described above is not a chance artifact of a single model: that networks of different builds, trained on different data, and perhaps different substrates (the brain among them), converge toward the same structure. It has been named the Platonic Representation Hypothesis; I develop it - with due caution - in a separate essay on the human-LLM resonance.

🎯 What’s next

We have built the bridge between text and numbers. Text breaks into tokens (subword - not words, not letters), and each token gets a vector from a trained embedding matrix - a vector that over time takes on a geometry of meaning (with due skepticism toward the famous “king - man + woman” analogy). We also added the encoding of position, without which the model would be blind to word order.

So we have a sequence of vectors, each aware of its place in the sentence. But the vectors still sit side by side in isolation - they do not “talk” to one another. And yet meaning depends on neighbors: “bank” means something different by a river than in finance, and in “the knife didn’t cut the bread because it was too blunt” the word “it” points to the knife - but change “blunt” to “stale” and it points to the bread instead. In the next stop of the series we reach the heart of the Transformer - the attention mechanism: how each token “looks around” at the others and takes from them what it needs to sharpen its own meaning.

If you would rather touch now than read on, the interactive page “Anatomy of an LLM” lets you see how text splits into tokens and how the vectors arrange themselves in space. And if what tempts you is what really emerges from the geometry described here - and why different minds seem to converge toward a shared structure of meaning - I wrote about that separately.