Anatomy of an LLM: the neuron and activation functions

🚀 Intro

In the overview post of this series we took a language model apart into seven stops - from the neuron to LoRA - in a way that let you walk the whole thing without a single formula. That was the bird’s-eye view. Now we come down to earth and begin the most serious part of the journey: into the mechanism, stop by stop, this time with the history, full formulas and concrete numbers from real models.

We start at the very bottom - at the building block out of which everything else is made: the single artificial neuron and that small “bend” on its output we call the activation function. It looks like the most boring element of the whole puzzle. And yet: without that bend even the largest model would collapse into a single multiplication and would learn nothing interesting, and the quiet evolution of these functions - from the sigmoid to SwiGLU - turned out to be one of the heroes behind the leap in quality of modern LLMs.

There is one more thing worth keeping in mind right at the entrance. The artificial neuron we are about to take apart is a drastic caricature of its biological original. And even so, out of billions of such caricatures something emerges that their formula in no way foretells. That tension - the simplicity of the parts, the unobviousness of the whole - will accompany us through the entire series.

💡 How to read this text. The main thread is descriptive and self-contained. I tuck the formulas into collapsible “for the curious” blocks - if you like math, expand them; if not, skip them with no loss to understanding. Every claim has a footnote to its source paper at the end.

📋 TL;DR

• The artificial neuron computes a weighted sum of its inputs, adds a bias and passes the result through an activation function. The idea is 80 years old (McCulloch and Pitts, 1943) and is a deliberate, heavy simplification of biology.
• The activation function is the only source of nonlinearity. Without it, a stack of even a thousand layers is mathematically equal to a single layer - that is, to an ordinary straight line.
• The universal approximation theorem says a network with a single hidden layer can approximate any continuous function. But it is a theorem about existence - it promises neither that such a network will be small nor that gradient descent will find it.
• ReLU (zero out negative numbers) displaced the sigmoid because it does not “saturate” and does not kill the gradient in deep networks. Its weakness - “dead” neurons - was patched by variants.
• Modern LLMs use smooth, gated activations: GPT - GELU, LLaMA and PaLM - SwiGLU, Gemma - GeGLU. This is not cosmetics: in benchmarks they gave the best quality.
• Everything below you can touch on the interactive page.

🧠 The neuron model: from McCulloch and Pitts’ threshold to today

The history of the artificial neuron begins with a question that sounds audacious today: can thinking be described by logic?

McCulloch and Pitts (1943). The first formal model of an artificial neuron is a binary threshold unit. It sums its inputs and “fires” (output 1) if the sum exceeds a fixed threshold; otherwise it stays silent (output 0). The neurophysiologist Warren McCulloch and the logician Walter Pitts showed that networks of such units can realize Boolean logic, and with the right connectivity are computationally universal¹. From that pair - a physician and a logician - came an idea that drives the entire industry today.

Rosenblatt’s perceptron (1958). Frank Rosenblatt went a step further and added what was missing: learning. His perceptron is a single-layer network of units with weighted inputs and a threshold activation, with an explicit rule for tuning the weights on examples². It is the first practically realized model of a neural network - and the first that “learned” from data instead of being programmed by hand.

Winter: the XOR limitation (1969). The enthusiasm was quickly cooled. In the book Perceptrons, Marvin Minsky and Seymour Papert proved that a single-layer perceptron represents only linearly separable functions and cannot compute even an operation as simple as XOR³. The result - often over-interpreted - contributed to the “winter” of neural-network research in the 1970s. The way out came only with multi-layer networks trained by backpropagation, which got around that limitation. (We will return to layers and training in the next post of the series.)

The modern neuron is a “softened”, smooth version of that threshold unit: instead of a hard “fire / stay silent” we have a continuous activation function. The rest is exactly the same idea - a weighted sum plus a bias.

📐 For the curious: the neuron equation

The neuron computes a weighted sum of inputs $x_i$ with weights $w_i$, adds a bias $b$, and passes the result $z$ through an activation function $\phi$ (phi):

$y = \phi\!\left(\sum_{i} w_i x_i + b\right) = \phi(\mathbf{w}\cdot\mathbf{x} + b)$

The notation $\mathbf{w}\cdot\mathbf{x}$ is the dot product - shorthand for the sum $w_1x_1 + w_2x_2 + \dots$. In the McCulloch-Pitts model $\phi$ was a step (a threshold); today it is a smooth function, which we will meet in a moment.

Before we go further, one honest caveat. The word “neuron” suggests we are reproducing the brain. We are not. The artificial neuron is a drastic idealization³, and it is worth knowing exactly what was thrown overboard:

• No spikes. Biological neurons communicate by voltage spikes (action potentials) - “all-or-nothing” events that encode information also in time and in the pattern of firing. The artificial neuron has a single continuous number, interpreted at most as an averaged rate.
• No memory. A real neuron is a dynamical system - its state depends on its history. A standard feedforward neuron is memoryless: its output depends only on the current input. Time is added later by the architecture (e.g. by autoregression in an LLM), not by the neuron itself.
• Linear summation. Real dendrites perform complex, nonlinear, location-dependent integration; here we have a simple weighted sum.
• Rigid synapses. The weights are fixed during a single pass; biological synapses have rich, time-dependent plasticity.

This is not a footnote for pedants. It is the first time in this series that the gulf is visible between the simplicity of the building block and the complexity of what grows out of it - and the reason why humility about the “thinking” of models is in order.

〰️ Why nonlinearity is necessary

Here comes a fact that decides everything - and that is easy to overlook. If a neuron only multiplied and summed, then composing layers would be composing linear transformations, and a composition of linear functions is again a linear function. In other words: a stack of a thousand layers without activations collapses, algebraically, to a single layer. The whole gigantic network would be able to model only linear relationships - that is, almost nothing of what interests us, because language, image and logic are not linear.

The rescue is precisely the activation function - a small “bend” inserted on the output of every neuron. It is what introduces nonlinearity, thanks to which the layers stop cancelling out and the network gains the power to model arbitrarily wavy relationships.

How much power? Surprisingly much - and it was proven formally. The universal approximation theorem (Cybenko, 1989) says that a feedforward network with a single hidden layer and a continuous sigmoidal nonlinearity can approximate any continuous function to any accuracy⁴. Two years later Hornik generalized the result: what is decisive is the multi-layer architecture itself, not the particular choice of activation function⁵. This sounds like a magical promise - one layer is enough for everything.

Except the promise has fine print, and the kind worth understanding, because it guards against naivety:

• It is a theorem about existence. It guarantees that an appropriate network exists - it says nothing about how large it must be or how to find it.
• The width can be exponential. For some functions a shallow (single-layer) network would have to have a number of neurons growing exponentially with the input dimension. Deep networks represent the same functions far more cheaply - this is the formal advantage of depth proven by Telgarsky⁶. That is why we build deep models, not wide ones.
• No guarantee of learnability. The theorem says a set of weights exists - it does not promise that gradient descent will reach it (the loss landscape is non-convex, full of local minima).

This distinction - “a solution exists” versus “we can find it” - is one of the most important motifs in all of machine learning. We will return to it when we talk about training.

📐 For the curious: what the theorem says exactly

Cybenko showed that finite linear combinations of compositions of a fixed sigmoidal function $\sigma$ (sigma; with affine transformations) are dense in the space of continuous functions on the unit hypercube $[0,1]^n$. Formally: for any continuous $g$ and any arbitrarily small error tolerance $\varepsilon>0$ (epsilon) there exists a sum

$G(\mathbf{x}) = \sum_{j=1}^{N} \alpha_j\,\sigma\!\left(\mathbf{w}_j\cdot\mathbf{x} + b_j\right), \qquad \text{such that}\quad |G(\mathbf{x}) - g(\mathbf{x})| < \varepsilon$

for all $\mathbf{x}$ in the domain⁴. The assumptions matter: the target function must be continuous, and the domain compact (closed and bounded). The theorem gives no rate - how many neurons $N$ are needed for a given $\varepsilon$ - and it is exactly this gap that the depth-advantage results fill⁶.

👉 Play with it: on the interactive page you have a neuron lab - drag the weights and inputs and watch the weighted sum change live, plus an activation-function chart on which you can switch between the shapes discussed below.

🔧 The zoo of activation functions and its evolution

If nonlinearity is necessary, the question remains: which bend should we choose? The answer evolved over decades, and each change solved a concrete, tangible problem of its predecessor. Let us trace that line.

Sigmoid and tanh: an elegance that chokes the gradient

The first popular activations were smooth and mathematically “pretty”. Sigmoid squeezes any number into the range $(0,1)$, and tanh into $(-1,1)$. The problem shows up in deep networks: for large input values both functions saturate - their graph goes flat and the slope (the derivative) tends to zero.

Why is this a catastrophe? Because learning consists of pushing the error signal backward through all the layers, multiplying along the way by these slopes. When each of them is a fraction close to zero, the product of several dozen such fractions becomes microscopic - this is the famous vanishing gradient. The effect: early layers learn at a snail’s pace, or not at all. This was one of the main walls that deep learning kept hitting before ReLU was popularized⁷.

📐 For the curious: formulas and derivatives of sigmoid and tanh

We write the sigmoid with the small Greek letter $\sigma$ (sigma), and the hyperbolic tangent as $\tanh$:

$\sigma(x) = \frac{1}{1+e^{-x}}, \qquad \sigma'(x) = \sigma(x)\big(1-\sigma(x)\big)$ $\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}, \qquad \tanh'(x) = 1 - \tanh^2(x)$

The maximum of the sigmoid’s derivative is a mere $0.25$ (at $x=0$), and tanh’s - $1$ (also at $x=0$). For large $|x|$ both derivatives tend to zero: hence the gradient vanishing in the product over many layers. Tanh has an extra advantage: it is centered around zero, which usually gives healthier learning dynamics than the sigmoid with its range $(0,1)$.

ReLU: brutally simple and therefore effective

The breakthrough turned out to be surprisingly primitive. ReLU (Rectified Linear Unit) is simply: “zero out negative numbers, pass positive ones through unchanged”. No exponentials, no saturation on the positive side - there the slope is exactly 1, so the gradient flows freely through even hundreds of layers. On top of that it is lightning-fast to compute and produces sparse representations (some neurons output a clean zero), which itself helps.

A curiosity: ReLU did not fall from the sky as an engineering trick. It emerged from theory - Nair and Hinton derived it by summing infinitely many copies of a binary unit with shifted thresholds, which yields a smooth approximation called softplus⁸. A year later Glorot, Bordes and Bengio showed empirically that rectifying neurons match or beat tanh, while producing the desired sparsity⁹.

ReLU has, however, its own Achilles’ heel: “dying” neurons (dying ReLU). For a negative input the derivative is 0, so if a neuron gets stuck with a negative sum for all examples, its gradient is permanently zero - the neuron stops learning and is practically dead. A whole family of patches arose for this ailment.

📐 For the curious: ReLU and its variants against "dead" neurons

$\mathrm{ReLU}(x) = \max(0, x), \qquad \mathrm{ReLU}'(x) = \begin{cases}1 & x>0\\ 0 & x<0\end{cases}$

(at zero the function is non-differentiable - in practice one takes the derivative as 0 or 1). The variants introduce a nonzero slope on the negative side, controlled by a parameter $\alpha$ (alpha), so the neuron never freezes completely - like Leaky ReLU¹⁰ with a fixed small $\alpha\approx 0.01$ or ELU:

$\mathrm{LeakyReLU}(x) = \begin{cases} x & x>0\\ \alpha x & x\le 0\end{cases} \qquad \mathrm{ELU}(x) = \begin{cases} x & x>0\\ \alpha(e^x - 1) & x\le 0\end{cases}$

In PReLU the slope $\alpha$ stops being a constant and becomes a learned parameter - the network itself decides how much to let negative values through. It was from PReLU that the loud 2015 result came: 4.94% top-5 error on ImageNet, the first to exceed human level (5.1%) on that benchmark¹¹. ELU additionally pushes the mean activation closer to zero (an effect similar to normalization), which speeds up learning¹².

GELU, Swish and gated GLUs: what really sits inside LLMs

Here we reach the functions that power today’s language models. ReLU has a sharp “elbow” at zero - it is non-smooth. Newer activations smooth out that corner and, more importantly, introduce subtle gating: instead of hard-passing or zeroing the input, they weight it softly depending on its value.

GELU (Gaussian Error Linear Unit), instead of asking “is $x$ positive?”, asks “how much does $x$ stand out against the noise?” and passes it through proportionally⁷. This is precisely the activation inside GPT-2 and GPT-3. Swish (also known as SiLU) is a related idea - a smooth interpolation between a line and ReLU, controlled by a single parameter¹³.

The biggest practical leap, however, came from gated linear units (GLU). The idea: split the signal into two linear projections, pass one of them through an activation and use it as a “gate” multiplying the other. From this lineage come GEGLU and SwiGLU - and this is not a curiosity but practice: in careful benchmarks (models matched by parameter count and compute) the GEGLU and SwiGLU variants gave the best quality of all the FFN activations tested¹⁴.

📐 For the curious: GELU, Swish and the GLU family

GELU weights the input by the probability that a normal variable does not exceed $x$ (where $\Phi$, capital “phi”, is the cumulative distribution function of the normal distribution)⁷:

$\mathrm{GELU}(x) = x\,\Phi(x) = x\cdot\tfrac{1}{2}\Big[1+\mathrm{erf}\!\big(x/\sqrt{2}\big)\Big] \approx 0.5\,x\Big(1 + \tanh\!\Big[\sqrt{2/\pi}\,\big(x + 0.044715\,x^3\big)\Big]\Big)$

Swish is $x$ multiplied by the sigmoid, with a parameter $\beta$ (beta) regulating the shape¹³:

$\mathrm{Swish}(x) = x\cdot\sigma(\beta x)$

For $\beta=1$ this is exactly SiLU; as $\beta\to\infty$, Swish tends to ReLU - hence “smooth interpolation”. The gated variants multiply element-wise (the symbol $\otimes$) the activated projection by the plain linear one¹⁴:

$\mathrm{GEGLU}(x) = \mathrm{GELU}(xW)\otimes xV, \qquad \mathrm{SwiGLU}(x) = \mathrm{Swish}(xW)\otimes xV$

The catch: GLUs have three weight matrices instead of two, so to keep the parameter count constant one reduces the hidden-layer width $d_{ff}$ by a factor of $\tfrac{2}{3}$¹⁴. That is why the mysterious "$\tfrac{2}{3}\cdot 4d$" shows up in the LLaMA descriptions.

🏭 What really sits inside GPT, LLaMA and Gemma

Theory is theory, but which function won in practice? Here is the hard fact, straight from the papers and the source code of the creators:

• GPT-2 / GPT-3 (OpenAI): GELU. Confirmed directly from the GPT-2 code - the gelu function is exactly the tanh approximation from Hendrycks and Gimpel’s paper¹⁵. For GPT-3, the use of GELU is taken to be consistent with GPT-2 (the GPT-3 paper itself does not state it explicitly¹⁶).
• PaLM (Google, 2022): SwiGLU - chosen because it “significantly improves quality over standard ReLU, GELU or Swish”¹⁷.
• LLaMA (Meta, 2023): SwiGLU - quoting the paper: “We replace the ReLU non-linearity by the SwiGLU activation function (…), we use a dimension of $\tfrac{2}{3}4d$ instead of $4d$ as in PaLM”¹⁸.
• Gemma (Google DeepMind, 2024): GeGLU - “The standard ReLU non-linearity is replaced by the GeGLU activation function”¹⁹.

The trend is clear: older OpenAI models stand on GELU, while the wave of modern, open LLMs (LLaMA, PaLM) switched to SwiGLU, whereas Gemma went for GeGLU. All of these choices are direct descendants of a single paper by Shazeer from 2020¹⁴. It is a good example of how a single, seemingly minor result - “multiply two projections instead of one” - seeps into an entire generation of models.

🎯 What’s next

We took the smallest building block fully apart: where it came from (McCulloch and Pitts’ threshold unit), why it needs nonlinearity (because otherwise a stack of layers collapses into a single line), what that nonlinearity promises and what it does not (universal approximation - existence, not feasibility) and which concrete “bend” won in real models (the smooth, gated GELU and SwiGLU).

Along the way we also saw the first crack in which the mystery of this series lives: our neuron is a caricature of biology, and yet out of its billions of copies grows something that the formula $y=\phi(\mathbf{w}\cdot\mathbf{x}+b)$ does not foretell with a single word. Understanding the building block takes nothing away from its remarkableness - it only shows how modest the raw material is.

In the next stop of the series we take a handful of these neurons and arrange them into layers and a network - and then ask the most important question: where do those famous “billions of parameters” actually come from, and how does training turn random noise into a model that understands language? That is where backpropagation, loss functions and scaling laws will appear.

If you would rather touch now instead of reading on, the interactive “Anatomy of an LLM” page lets you drag the neuron’s weights and switch activation functions live. And if you are curious about what actually emerges from this machinery once the building blocks number in the billions - I wrote about that separately.

Footnotes

1. W. S. McCulloch, W. H. Pitts (1943). A Logical Calculus of the Ideas Immanent in Nervous Activity. The Bulletin of Mathematical Biophysics 5(4):115-133. https://doi.org/10.1007/BF02478259 ↩

2. F. Rosenblatt (1958). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review 65(6):386-408. https://doi.org/10.1037/h0042519 ↩

3. M. Minsky, S. Papert, Perceptrons (MIT Press, 1969) - the proof of the XOR limitation and the classic characterization of the artificial neuron as an idealization of the biological one (facts independently confirmed in the universal-approximation works). ↩ ↩²

4. G. Cybenko (1989). Approximation by Superpositions of a Sigmoidal Function. Mathematics of Control, Signals and Systems 2:303-314. https://doi.org/10.1007/BF02551274 ↩ ↩²

5. K. Hornik (1991). Approximation Capabilities of Multilayer Feedforward Networks. Neural Networks 4(2):251-257. https://doi.org/10.1016/0893-6080(91)90009-T ↩

6. M. Telgarsky (2016). Benefits of Depth in Neural Networks. COLT 2016, PMLR 49:1517-1539. https://arxiv.org/abs/1602.04485 ↩ ↩²

7. D. Hendrycks, K. Gimpel (2016). Gaussian Error Linear Units (GELUs). arXiv:1606.08415. https://arxiv.org/abs/1606.08415 ↩ ↩² ↩³

8. V. Nair, G. E. Hinton (2010). Rectified Linear Units Improve Restricted Boltzmann Machines. ICML-10, pp. 807-814. https://www.cs.toronto.edu/~fritz/absps/reluICML.pdf ↩

9. X. Glorot, A. Bordes, Y. Bengio (2011). Deep Sparse Rectifier Neural Networks. AISTATS, PMLR 15:315-323. https://proceedings.mlr.press/v15/glorot11a.html ↩

10. A. L. Maas, A. Y. Hannun, A. Y. Ng (2013). Rectifier Nonlinearities Improve Neural Network Acoustic Models. ICML Workshop. https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf ↩

11. K. He, X. Zhang, S. Ren, J. Sun (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. arXiv:1502.01852. https://arxiv.org/abs/1502.01852 ↩

12. D.-A. Clevert, T. Unterthiner, S. Hochreiter (2015). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv:1511.07289 (ICLR 2016). https://arxiv.org/abs/1511.07289 ↩

13. P. Ramachandran, B. Zoph, Q. V. Le (2017). Searching for Activation Functions (Swish). arXiv:1710.05941. https://arxiv.org/abs/1710.05941 ↩ ↩²

14. N. Shazeer (2020). GLU Variants Improve Transformer. arXiv:2002.05202. https://arxiv.org/abs/2002.05202 ↩ ↩² ↩³ ↩⁴

15. OpenAI. GPT-2 source code, src/model.py, the gelu function. https://github.com/openai/gpt-2/blob/master/src/model.py ↩

16. T. B. Brown et al. (2020). Language Models are Few-Shot Learners (GPT-3). arXiv:2005.14165. https://arxiv.org/abs/2005.14165 ↩

17. A. Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311. https://arxiv.org/abs/2204.02311 ↩

18. H. Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971. https://arxiv.org/abs/2302.13971 ↩

19. Gemma Team, Google DeepMind (2024). Gemma: Open Models Based on Gemini Research and Technology. arXiv:2403.08295. https://arxiv.org/abs/2403.08295 ↩