From a single neuron, through weights and activation functions, all the way to the attention mechanism and fine-tuning with LoRA. Short, with math and interactive diagrams.
An entire neural network is made of millions of copies of one very simple element. An artificial neuron takes a few input numbers x, multiplies each by its assigned weight w, sums the results, adds a bias b (an offset), and finally passes the result through an activation function f.
Mathematically, the whole neuron is a single equation. First we compute the weighted sum z, then we pass it through the function:
The notation w·x is the dot product - the same as the sum w₁x₁ + w₂x₂ + …, just shorter. The weight tells the neuron how important a given input is (large positive = amplifies, negative = suppresses), and the bias shifts the firing threshold.
If neurons only summed and multiplied, the entire network - even with a thousand layers - would be equivalent to a single matrix multiplication, i.e. a straight line. The activation function introduces nonlinearity, which lets the network model complex, curved relationships (language, images, logic).
| Function | Formula | Range | Where it's used |
|---|---|---|---|
| ReLU | max(0, z) | [0, ∞) | hidden layers (classic, fast) |
| Sigmoid | 1/(1+e⁻ᶻ) | (0, 1) | probability, gates |
| tanh | (eᶻ−e⁻ᶻ)/(eᶻ+e⁻ᶻ) | (−1, 1) | recurrent networks |
| GELU | z·Φ(z) | ≈[−0.17, ∞) | modern Transformers (GPT, BERT) |
A single neuron is weak. Power only comes from arranging them into layers and connecting layers into a network. The outputs of one layer become the inputs of the next. The signal flows from left (input) to right (output) - this is the so-called forward pass.
Instead of computing each neuron separately, we write the whole layer as a single matrix multiplication. If a layer has a weight matrix W and a bias vector b, then:
This is why LLM training happens on graphics cards (GPUs) - they are designed for lightning-fast multiplication of huge matrices. All of the model's "intelligence" sits in the numbers inside these matrices W.
When you hear "the model has 7B (7 billion) parameters", it refers precisely to the number of all weights and biases in the network. These are the numbers the model "learns" during training, gradually correcting them so its predictions get better and better.
Example: a layer taking 1000 inputs and producing 1000 outputs already has 1,000,000 + 1000 ≈ a million parameters. An LLM has hundreds of such layers - hence the billions.
Weights are not set by hand. The model starts with random numbers and improves them in a loop:
Every training step nudges the weights and usually lowers the loss a little. How fast (and whether) the loss falls depends on the learning rate η. Play with the slider to see three regimes: too slow, just right, and divergent.
Given a fixed compute budget, how many parameters versus how many training tokens should you pick? The Chinchilla paper (DeepMind, 2022) showed the sweet spot is about 20 tokens per parameter - earlier models were often undertrained.
The network doesn't understand letters. Text is first split into tokens (chunks of words), and each token is turned into a vector of numbers - an embedding. These are the "coordinates" of a word in a high-dimensional space of meaning.
The beauty of embeddings is that geometry carries meaning. The words "cat" and "dog" lie close together; relationships are vectors too - the classic example:
Type any text and watch how it would split into tokens. Common words stay whole; rarer and longer ones break into pieces (## marks a word continuation). This split is illustrative - real tokenizers (BPE) learn their merges from a dataset.
Since words are vectors, you can add and subtract them. Pick three words and see which word the result lands closest to. (A toy 2D space: the horizontal axis ~ gender, the vertical axis ~ status. Real embeddings have thousands of dimensions, and such analogies can be unreliable.)
This is the breakthrough that made today's LLMs possible. Attention lets every word "look around" the entire sentence and decide which other words matter most to it. In the sentence "The cat didn't eat the fish because it was spoiled" - attention links "it" to "fish", not to "cat".
From each token's embedding, the model creates three vectors by multiplying it by three learned matrices:
Matching the Query against each Key (dot product) gives a score; after scaling and passing through softmax we get attention weights that sum to 1. The result is a weighted sum of the values V:
Dividing by √dₖ (the square root of the dimension) stabilizes the values so that softmax doesn't "saturate" at large numbers. Softmax turns raw scores into attention percentages.
↑ The values are illustrative. The darker/longer the bar, the more "attention" the selected word devotes to a given word in the sentence.
The bars above show a single row of this matrix. Here you see all of it: each row is a query word, each column a key word, and a cell's brightness says how much attention the former pays the latter. This visualizes the matrix softmax(QKᵀ/√d) - the same one we computed above.
↑ The numbers are the percentage of attention (each row sums to 100%). Notice the bright cell "it/spoiled → fish" - exactly the link the attention mechanism captures.
A single attention "head" looks at one type of relationship. The model runs dozens of them in parallel - one tracks grammar, another references, another style - and their results are combined. Hence the name multi-head attention.
An LLM is a stack of many identical Transformer blocks (e.g. 32, 80, or even over 100). Each block has two main parts: an attention layer and an ordinary feed-forward network (FFN). They are held together by two tricks that make training very deep networks possible:
At the very end of the stack, the final layer turns the vectors back into probabilities for the next token - and it's from these that the next word of the answer is "sampled".
The final layer yields a probability distribution over all tokens. How we sample from it is set by two knobs: temperature (sharpens or flattens the distribution) and top-p (cuts off the tail of unlikely words). Play with them:
↑ Grey bars show the full post-temperature distribution; red ones are the tokens that survived the top-p cut and that the model actually samples from (✕ = cut).
Full fine-tuning of a 70B model means updating all 70 billion parameters - this requires huge graphics cards and a lot of money. LoRA (Low-Rank Adaptation) is a clever trick: we freeze the original weights and train only a tiny "add-on".
The weight change needed to specialize a model is "low-rank" - it can be written as the product of two thin matrices. Instead of training the whole large update matrix ΔW (with dimensions d×d), we train only two skinny matrices A and B:
Where r is the rank (e.g. 8 or 16) - the bottleneck that determines the number of new parameters, and α is the scale of the adapter's influence. W₀ stays untouched.
Let's compare the full update with LoRA for a matrix d = 4096 and rank r = 8:
| Approach | Parameter-count formula | Result (d=4096, r=8) |
|---|---|---|
| Full fine-tuning (ΔW) | d × d | ≈ 16,800,000 |
| LoRA (B + A) | d × r + r × d = 2·d·r | ≈ 65,500 |
| Savings | d / (2r) | ~256× less |
The token x passes through the frozen main path W₀x and through the cheap LoRA "side track" - and the results are summed. That's all.
The rank r is LoRA's one real knob: the smaller it is, the fewer new parameters (and the cheaper), but the less "capacity" for new knowledge. See how r translates into the number of trained parameters (the bar scale is logarithmic):