Anatomy of an LLM: the Transformer block

🚀 Intro

In the previous stop of this series - one of the branches of the overview post “Anatomy of an LLM” - we took the attention mechanism apart: we saw how a token forms a query, a key, and a value, how it compares itself to the others, and how it mixes into itself the information from its most relevant neighbors. But we ended on an important caveat: attention is just one building block. On its own it is not yet a model.

In this post we assemble the whole machine from those blocks. We will see how the attention layer joins with a processing network into one repeatable Transformer block, how two simple ideas - residual connections and normalization - let us stack those blocks into towers of dozens of layers without wrecking the training, and how, at the very end, the model turns its internal vector of numbers back into the word you are reading.

Along the way we also answer a question that is usually left hanging: since there are different ways to build such models, why did today’s LLMs - from ChatGPT to Claude - settle on just one of them? And as always in this series: the blueprint itself will turn out to be surprisingly simple and regular, while what grows out of it at sufficient scale remains not fully predictable.

💡 How to read this. The main thread is descriptive and self-contained. I tuck the formulas into expandable “for the curious” blocks - if you like math, open them; if not, skip them with no loss to understanding. Every claim has a footnote to its source paper at the end.

📋 TL;DR

• Three families, one winner. From the original Transformer grew three architectures: encoder-decoder (T5), encoder-only (BERT) and decoder-only (GPT). Today’s generative LLMs are almost always decoders - because the goal “predict the next token” is naturally generative, simple, and scales beautifully.
• Residual connections = a highway for the gradient. Each sub-layer does not replace its input but adds a correction to it. Thanks to this “shortcut,” the gradient reaches the deepest layers and we can train towers of hundreds of layers.
• Normalization stabilizes. LayerNorm (and in newer models the cheaper RMSNorm) brings activations to a common scale. A small decision - whether to normalize before the sub-layer (Pre-LN) or after (Post-LN) - decides whether training needs a “warm-up” or not.
• The FFN is a store of knowledge. After attention, each token passes through a small feed-forward network (FFN), which expands the vector fourfold and squeezes it back. About 2/3 of a block’s parameters live in it - and probably most of the model’s “knowledge.”
• From vector to word. The final vector of each token is projected onto the whole vocabulary (logits), and the softmax turns those logits into a probability distribution over the next word.
• Decoding is no afterthought. How we pick a token from that distribution - greedily, or by sampling with temperature and nucleus sampling - decides whether the text is dull, unhinged, or just right.
• Mixture of Experts. The newest direction: instead of one FFN the model has many “experts” and fires only a few of them per token. A huge parameter count at a constant compute cost.

👉 Play with it: on the interactive page you have a section on the Transformer block - see how the layers stack up with residual connections and normalization.

🏛️ Three families: why an LLM is a “decoder”

The original Transformer (Vaswani et al., 2017) was designed for machine translation and had an encoder-decoder shape: two stacks of six layers each. The encoder read the whole source sentence, the decoder produced the translation, looking back into the encoder through cross-attention¹. But from this single architecture three families of models eventually grew - depending on which half you keep and how you train it.

• Encoder-only - the flagship example is BERT (Devlin et al., 2018). Just the encoder stack with bidirectional attention: each token sees what is to its left and right at once. You train it by masking random words and asking it to guess them (Masked Language Modeling). Excellent for understanding text - classification, search, analysis - but by nature it does not generate text word by word².
• Encoder-decoder - e.g. T5 (Raffel et al., 2019), which frames every NLP task as “text in → text out”³. It keeps both halves of the original.
• Decoder-only - GPT (Radford et al., 2018/2019) and nearly everything we today call a “large language model.” Just the decoder stack with masked (causal) attention, which we met in the previous post: each token sees only the past. You train it as a plain autoregressive language model - “predict the next token”⁴⁵.

Why did decoders come to dominate the world of generative LLMs? Several things lined up. The autoregressive goal - “guess the next word” - is naturally generative (the model learns exactly what it later does) and self-supervised: it needs no labels, raw text is its own answer, so you can feed it the entire internet. The architecture is also simpler - one stack, no cross-attention - and allows efficient generation with the KV-cache discussed previously. The GPT approach simply turned out to scale best with model size and amount of data⁶.

Here, though, it is only fair to plant a flag of caution: “decoder is better” is an empirical observation, well grounded in engineering and confirmed by the practice of recent years, not a theorem with a formal proof. It just so happened that this road went the furthest.

🔗 Residual connections: a highway for the gradient

When we try to build very deep networks - tens, hundreds of layers - an old enemy from earlier posts appears: the vanishing gradient. The learning signal, traveling back through many layers, weakens so much that the deepest layers barely learn. For years this was a ceiling that the depth of networks kept hitting.

The breakthrough came from image recognition. ResNet (He et al., 2015) introduced an idea so simple it is almost surprising it had to be waited for: instead of asking a layer to learn the whole desired transformation from scratch, ask it to learn only a correction to what it already received as input. A layer’s output is not “a new vector” but “the input vector plus whatever the layer wanted to add.” This seemingly cosmetic move made it possible to train networks 152 layers deep, and in experiments even a thousand layers deep⁷.

Why does it work? Because that “plus the input” creates a shortcut - literally a highway down which the gradient can drive backward through the whole network, bypassing all the costly transformations along the way. Even if a layer in the middle “throttles” the signal, the identity bypass stays open. In the Transformer every sub-layer - both attention and the FFN - is wrapped in such a residual connection. They are the backbone that makes it possible to stack dozens of layers one on another at all.

There is one quiet technical condition here: since we add the input to a sub-layer’s output, both must have the same size. That is why all sub-layers and embeddings in the Transformer share one common dimension $d_{\text{model}}$ (read “d-model”) - this is why that single number runs through the whole architecture¹.

📐 For the curious: the math of the residual connection

In ResNet, instead of teaching a layer the target transformation $H(x)$ directly, we teach it the residual (hence “residual”) $F(x) = H(x) - x$, then reassemble the output:

$y = F(x) + x$

When we compute the gradient backward, the derivative of this sum contains an identity term (from the "$+x$"), which passes the learning signal onward regardless of what $F$ does. It is exactly that one in the derivative that is the “highway.”

In the original Transformer (the Post-LN variant) the full output of a sub-layer is:

$\text{LayerNorm}\big(x + \text{Sublayer}(x)\big)$

where $\text{Sublayer}$ is either attention or the FFN¹. For the addition of $x$ to be feasible at all, $x$ and $\text{Sublayer}(x)$ must have an identical dimension $d_{\text{model}}$.

📏 Normalization: a common scale for all

The second pillar, without which deep towers would not train, is normalization. The problem is this: as the signal passes through many layers, activation values tend to drift - now huge, now vanishingly small. Such a mismatch of scale destabilizes learning. Normalization acts like a constant “resetting of the measure”: it brings the vector back to a standardized range before passing it on.

The standard in the Transformer is LayerNorm (Ba, Kiros, Hinton, 2016). It computes the mean and the spread within a single token, across all of its features, and standardizes the vector on that basis⁸. The key thing here is the difference from an earlier idea called BatchNorm, which computed statistics across the whole batch of examples. LayerNorm looks only at a single example, so it is independent of the size and composition of the batch and works identically in training and in use - which makes it a natural choice for sequences of variable length.

A subtle but consequential question arises, though: should we normalize before the sub-layer, or after it?

• Post-LN (as in the original): first attention or FFN, then the addition of the input, and normalization at the end.
• Pre-LN (as in modern models): first normalize the input, only then the sub-layer and the addition.

Xiong et al. (2020) showed this is not cosmetic. In Post-LN the gradients at layers near the output are very large at the start of training, which makes it unstable and forces a “warm-up” of the learning rate - slowly ramping it up at the beginning. In Pre-LN the gradients are well-conditioned from the outset, so the warm-up can be skipped and training is more stable⁹. That is why today’s large models - GPT-2, GPT-3, LLaMA - use Pre-LN.

Newer models go a step further and reach for RMSNorm (Zhang and Sennrich, 2019). It is a simplified LayerNorm: it drops the subtraction of the mean (centering), keeping only the scaling. The result? Roughly the same quality, but computation cheaper by several to several dozen percent¹⁰. LLaMA and many contemporary models use precisely RMSNorm in a pre-normalization setup.

📐 For the curious: LayerNorm and RMSNorm in formulas

LayerNorm. For one token’s feature vector $x \in \mathbb{R}^{d}$ we first compute the mean and variance over its coordinates:

$\mu = \frac{1}{d}\sum_{i=1}^{d} x_i, \qquad \sigma^2 = \frac{1}{d}\sum_{i=1}^{d}(x_i - \mu)^2$

then standardize and rescale with learned parameters:

$\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$

Here $\mu$ (mu) is the mean, $\sigma^2$ (sigma squared) is the variance, a measure of spread, and $\sigma$ (sigma) without the square is the standard deviation. The tiny $\epsilon$ (epsilon) guards against division by zero. The vectors $\gamma$ (gamma) and $\beta$ (beta) are learned scale and shift parameters - they let the network undo the normalization if it happens not to need it. The symbol $\odot$ denotes element-wise multiplication⁸.

RMSNorm throws out the subtraction of the mean and simply divides by the square root of the mean of squares (hence the name - root mean square):

$\text{RMSNorm}(x) = \frac{x}{\sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}} \odot \gamma$

Fewer operations, no computing of the mean - hence the saving in time at comparable quality¹⁰.

🧮 Feed-Forward Network: a store of knowledge

After the attention sub-layer, a second sub-layer waits in each block: the feed-forward network (FFN; also called the MLP). It is a simple, two-layer network of the kind we met at the very start of the series - but here it has a crucial role.

It works on an “expand and compress” scheme. First it takes the token’s vector and projects it into a space four times larger (in the original: from 512 dimensions to 2048), passes it through a nonlinearity, and then squeezes it back to the output dimension¹. This momentarily “inflated” space gives the network far more room to represent and mix features - as if, for a moment, it laid all its tools out on a bigger table to pick the right ones, and then packed them away again. The 4× factor is a convention repeated in most models (in GPT-3 that hidden layer has over 49 thousand dimensions).

And here lurks something that surprises many: it is not attention, but the FFN, that usually swallows most of a block’s parameters - about two thirds. If attention is the mechanism of communication between tokens (who exchanges information with whom), then the FFN is the mechanism of processing what a given token has gathered - and probably the model’s main store of knowledge. More and more research suggests that memorized facts and patterns sit in the FFN’s weights. (This is an approximate ratio - the exact one depends on the architecture.)

The nonlinearity inside the FFN has also changed over time. The original used ReLU, but BERT and GPT moved to the smoother GELU¹¹, and many of today’s models to SwiGLU (Shazeer, 2020), a gated variant in which one projection “regulates the flow” of another¹². LLaMA uses precisely SwiGLU - and so as not to inflate the parameter count despite three matrices instead of two, it shrinks the hidden dimension to about two thirds of what a plain 4× would give¹³.

📐 For the curious: FFN and SwiGLU in formulas

The classic FFN is two linear layers with a nonlinearity $\max(0, \cdot)$ (i.e. ReLU) between them:

$\text{FFN}(x) = \max(0,\, xW_1 + b_1)\,W_2 + b_2$

The first matrix $W_1$ expands the dimension to $d_{ff}$ (read “d-f-f”), usually $4 \times d_{\text{model}}$; the second matrix $W_2$ compresses it back¹.

Where does “2/3 of the parameters” come from? Two FFN matrices of size $d_{\text{model}} \times 4d_{\text{model}}$ total about $8\,d_{\text{model}}^2$ weights, while the four attention projections (queries, keys, values, output) total about $4\,d_{\text{model}}^2$. Hence the proportion of roughly $8:4$, that is $2/3$ in the FFN’s favor.

SwiGLU replaces the single nonlinearity with a gate - the product of two separate projections, one of which passes through the Swish function:

$\text{SwiGLU}(x) = \big(\text{Swish}(xW_1) \odot xW_3\big)\,W_2$

The symbol $\odot$ is element-wise multiplication - it is what realizes the “gating”: the values of one branch regulate how much to let through from the other. Three matrices instead of two would increase the parameter count, so LLaMA compensates by shrinking the hidden dimension to $\frac{2}{3}\cdot 4d$¹²¹³.

🎰 From vector to word: the output layer

The token has passed through the whole stack of blocks - attention, FFN, residuals, normalization, layer after layer. At the output of the last block it is still what it was the whole way: a vector of numbers of dimension $d_{\text{model}}$. We still have to turn it into something we understand - into a predicted word.

This is done by the language head (the LM head or unembedding - the inverse of turning a token into a vector from the third post of the series). It is one large matrix that projects the token’s vector onto the whole vocabulary: for each of the tens of thousands of possible tokens it computes one number - the so-called logit, a raw “match score.” The higher the logit, the more the model leans toward that token.

Logits are not yet probabilities, though - they are arbitrary numbers, positive and negative. An old friend turns them into a proper probability distribution: the softmax. It takes all the logits, raises them to a power (which makes them positive and sharpens the differences) and normalizes them so they sum to one. The result is a distribution: “with what probability is the next word each token in the vocabulary.”

A thrifty curiosity: in many models (e.g. GPT-2) the same matrix serves both to turn a token into a vector at the input and to project back onto the vocabulary at the output - this is so-called weight tying, which saves a huge number of parameters. (It is not universal, though - LLaMA, for instance, keeps separate matrices.)

📐 For the curious: logits and softmax at the output

Let $h \in \mathbb{R}^{d_{\text{model}}}$ be the token’s vector after the last block. The language head projects it with the unembedding matrix $W_U$ onto a vector of logits of length equal to the vocabulary size $|V|$:

$z = h\,W_U$

The softmax turns the logits $z$ into a probability distribution over the vocabulary:

$P(\text{token}_i \mid \text{context}) = \frac{\exp(z_i)}{\sum_{j=1}^{|V|} \exp(z_j)}$

The denominator (the sum over all tokens of the vocabulary, denoted by the symbol $\Sigma$ - capital sigma) guarantees that the probabilities sum to one. Under weight tying, the matrix $W_U$ is simply the transpose of the input embedding matrix.

🎲 Decoding: how a word is really chosen

We have a probability distribution over the whole vocabulary. You would think: pick the most probable token and be done. Except how we pick the next token from that distribution - called the decoding strategy - turns out to be one of the most important knobs of quality. The interactive page sums it up in one word, “sampling,” but a whole family of methods hides behind it.

• Greedy decoding - always take the token with the highest probability. Deterministic and safe, but prone to dullness and to looping in repetitions¹⁴.
• Beam search - keep $k$ best partial sentences at once and expand them in parallel. Excellent for tasks with one “correct” answer, like translation, but for open, creative generation it gives text that is unnaturally smoothed and repetitive¹⁴.
• Temperature $T$ - the “creativity” knob. It scales the logits just before the softmax: $T$ below one sharpens the distribution (the model turns conservative, betting on sure things), $T$ above one flattens it (more variety, but also more risk), and $T$ approaching zero is the limit of greedy decoding.
• Top-k sampling (Fan et al., 2018) - cut the distribution down to the $k$ most probable tokens, renormalize them, and only then sample. It chops off the long tail of nonsensical words¹⁵.
• Top-p / nucleus sampling (Holtzman et al., 2019) - a smarter cut: take the smallest set of tokens whose cumulative probability exceeds a threshold $p$ (e.g. 0.9), and sample from it. The set is dynamic - wide when the model is uncertain, narrow when it is confident¹⁶.

In practice, today’s models most often combine temperature with nucleus sampling. That is why the same request asked twice can give two different answers - and why the “temperature” slider in an API is no ornament but a real knob between boredom and chaos.

📐 For the curious: temperature and nucleus sampling

Temperature divides the logits by $T$ before the softmax:

$P_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

For $T < 1$ the differences between logits grow (a sharper distribution), for $T > 1$ they shrink (a flatter distribution), and $T \to 0$ gives, in the limit, pure greedy decoding.

Nucleus sampling picks the smallest set of tokens $V^{(p)}$ such that their cumulative probability exceeds the threshold $p$:

$\sum_{i \in V^{(p)}} P_i \ge p$

It then zeroes out all tokens outside that set, renormalizes the rest, and samples from it. The key advantage over top-k: the size of the set adapts to the model’s confidence instead of being fixed¹⁶.

📊 Numbers from real models

Enough abstraction - let us see how these same blocks line up in concrete, well-known models. The same handful of numbers - the number of layers, the model dimension, the number of attention heads - describes everything from the modest original to the giants:

Model	Layers	$d_{\text{model}}$	Heads	$d_{ff}$	Parameters	Source
Transformer (base)	6 + 6	512	8	2048	~65M	Vaswani 20171
Transformer (big)	6 + 6	1024	16	4096	~213M	Vaswani 20171
GPT-3	96	12,288	96	49,152	175B	Brown 20206
LLaMA 7B	32	4096	32	~11,008	6.7B	Touvron 202313
LLaMA 65B	80	8192	64	-	65.2B	Touvron 202313

You can see at once how depth and width scale: from 6 layers and dimension 512 in the original to 96 layers and a dimension exceeding 12 thousand in GPT-3. For the record - GPT-3 was trained on batches of 3.2 million tokens, and LLaMA on a trillion tokens and more, with RMSNorm, SwiGLU and RoPE all included⁶¹³. (Some $d_{ff}$ values for the larger LLaMA models were not stated explicitly in the source paper, so I leave them blank rather than guess.)

🧩 Mixture of Experts: a modern direction

Finally, an architectural novelty that breaks the quiet rule holding through this whole series: that the same parameters process every input. Mixture of Experts (MoE) says: but maybe not all of them?

The idea is this. Instead of one FFN in a block we place many of them - call them “experts.” For each token a small gating network selects only a few of them, and only those experts do the work. The rest sleep. The effect is paradoxical and beautiful: the model can have a gigantic parameter count (that whole army of experts), yet the cost of processing one token stays constant, because only a handful ever fire. This is so-called sparse activation.

The idea is not new - Shazeer et al. (2017) built an MoE layer with thousands of sub-networks and a trained gate, reaching 137 billion parameters¹⁷. Switch Transformers (Fedus et al., 2021) simplified the routing so each token goes to one expert, and crossed a trillion parameters, speeding up pretraining several-fold over dense counterparts along the way¹⁸. In practice you swap the FFN sub-layer for an MoE layer and leave attention dense. More and more of today’s leading models are precisely MoE architectures - it is one of the main ways the industry keeps growing without paying a linearly rising bill for every token.

🌗 Regularity, from which the irregular grows

Let us pause for a moment, because we have just seen something worth naming outright. The whole Transformer block - the heart of every LLM today - is astonishingly regular. It is one and the same pattern: attention, add the input, normalize; FFN, add the input, normalize. Repeated dozens of times, like the same beat tapped out through an entire piece of music. There is no architectural magic in it - an engineer understands each block on its own and every connection between them.

And yet from that monotonous regularity, once you stack it deep enough and feed it a large enough corpus, there emerges something the blueprint does not contain: the ability to reason, translate, write code, hold a conversation. No one designed a layer “for understanding irony” or a block “for mathematical proofs.” There is only the same repeating beat and the pressure from earlier posts - “predict the next token as well as you can.” The rest arranges itself inside, in a way we still cannot fully trace.

And here we have perhaps the purest showing of the riddle that has recurred in this series since the very first post about the neuron-as-caricature. For we usually suspect that richness of behavior must come from richness of construction - that somewhere inside there is specialized machinery “for understanding.” And the deeper we took the Transformer block apart, the fewer such places we found: in their stead, the same beat repeated all the way down. The simpler and more repetitive the blueprint turns out to be, the more it puzzles us that all that richness comes from it. The mystery did not vanish under our scalpel - it merely shifted from the question “what is this built from” toward the harder “how does something so irregular come from something so regular.”

With that second question - and with a serious hypothesis that the answer may reach beyond this architecture itself, toward what different thinking substrates (the brain included) seem to have in common - I deal separately, in the essay on human-LLM resonance.

🎯 What’s next

We have assembled the whole model. We now know how the attention layer and the FFN join into one Transformer block, how residual connections and normalization let us stack dozens of such blocks, why today’s LLMs are decoders, and how, at the very end, logits and the softmax turn the internal vector into a concrete word - picked by one of the decoding strategies. We also saw how the same blocks scale from the original to the giants, and how Mixture of Experts pushes the limits of size.

But there is one more stop - and a very practical one. Training such a model from scratch costs millions. What, though, when you have a ready model and want only to specialize it - to teach it a legal style, your own domain knowledge, a particular tone? Fine-tuning all of its billions of parameters would be absurdly expensive. In the next, series-closing post we will meet LoRA and the rest of the family of “thrifty fine-tuning” methods (PEFT) - clever tricks that let you adapt a giant model by moving only a fraction of its weights.

If you would rather touch now than read on, the interactive page “Anatomy of an LLM” lets you see how the layers stack up with residual connections and normalization. And if what really emerges from this regular machinery pulls at you - and why a simple, repetitive blueprint gives rise to abilities no one wrote into it - I wrote about that separately.

Footnotes

1. A. Vaswani et al. (2017). Attention Is All You Need (encoder-decoder architecture, residual connections, 4× FFN, base and big model dimensions). NeurIPS. arXiv:1706.03762. https://arxiv.org/abs/1706.03762 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

2. J. Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (encoder-only, Masked Language Modeling). NAACL. arXiv:1810.04805. https://arxiv.org/abs/1810.04805 ↩

3. C. Raffel et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5, encoder-decoder, “text-to-text” framing). JMLR. arXiv:1910.10683. https://arxiv.org/abs/1910.10683 ↩

4. A. Radford et al. (2018). Improving Language Understanding by Generative Pre-Training (GPT, decoder-only, autoregressive language model). OpenAI. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf ↩

5. A. Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2, Pre-LN). OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf ↩

6. T. Brown et al. (2020). Language Models are Few-Shot Learners (GPT-3 175B, scaling of the decoder approach, architecture numbers). NeurIPS. arXiv:2005.14165. https://arxiv.org/abs/2005.14165 ↩ ↩² ↩³

7. K. He et al. (2016). Deep Residual Learning for Image Recognition (ResNet, residual connections, depth up to 152/1000 layers). CVPR. arXiv:1512.03385. https://arxiv.org/abs/1512.03385 ↩

8. J. L. Ba, J. R. Kiros, G. E. Hinton (2016). Layer Normalization (LayerNorm, difference from BatchNorm). arXiv:1607.06450. https://arxiv.org/abs/1607.06450 ↩ ↩²

9. R. Xiong et al. (2020). On Layer Normalization in the Transformer Architecture (Post-LN vs Pre-LN, warm-up). ICML. arXiv:2002.04745. https://arxiv.org/abs/2002.04745 ↩

10. B. Zhang, R. Sennrich (2019). Root Mean Square Layer Normalization (RMSNorm, saving in time). NeurIPS. arXiv:1910.07467. https://arxiv.org/abs/1910.07467 ↩ ↩²

11. D. Hendrycks, K. Gimpel (2016). Gaussian Error Linear Units (GELUs) (GELU activation in BERT and GPT). arXiv:1606.08415. https://arxiv.org/abs/1606.08415 ↩

12. N. Shazeer (2020). GLU Variants Improve Transformer (SwiGLU, gating in the FFN). arXiv:2002.05202. https://arxiv.org/abs/2002.05202 ↩ ↩²

13. H. Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models (Pre-LN/RMSNorm, SwiGLU with the 2/3·4d dimension, RoPE, full architecture table). arXiv:2302.13971. https://arxiv.org/abs/2302.13971 ↩ ↩² ↩³ ↩⁴ ↩⁵

14. Greedy and beam search as the backdrop to the text-degeneration problem are discussed in: A. Holtzman et al. (2020). The Curious Case of Neural Text Degeneration. ICLR. arXiv:1904.09751. https://arxiv.org/abs/1904.09751 ↩ ↩²

15. A. Fan, M. Lewis, Y. Dauphin (2018). Hierarchical Neural Story Generation (top-k sampling). ACL. arXiv:1805.04833. https://arxiv.org/abs/1805.04833 ↩

16. A. Holtzman et al. (2020). The Curious Case of Neural Text Degeneration (nucleus / top-p sampling, degeneration under greedy and beam search). ICLR. arXiv:1904.09751. https://arxiv.org/abs/1904.09751 ↩ ↩²

17. N. Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (MoE layer, sparse activation, up to 137 billion parameters). ICLR. arXiv:1701.06538. https://arxiv.org/abs/1701.06538 ↩

18. W. Fedus, B. Zoph, N. Shazeer (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (routing to a single expert, models up to a trillion parameters). JMLR. arXiv:2101.03961. https://arxiv.org/abs/2101.03961 ↩