Anatomy of an LLM: LoRA and fine-tuning

🚀 Intro

At the previous stop on this series - one of the branches of the overview post “Anatomy of an LLM” - we assembled the whole machine from its parts: the Transformer block, a stack of dozens of such blocks, the output layer that turns a vector of numbers back into a word. We have a finished model. The trouble is that training it from scratch cost millions of dollars and devoured a trillion tokens - and now we would only like to specialize it a little. Teach it legalese, our own domain knowledge, a particular customer-support tone.

The naive answer is: fine-tune it. Take the finished model and keep training, on your data. The problem is that “fine-tune” means updating all of its billions of parameters - and that requires almost as much hardware as training from scratch. For a model on the order of 70 billion weights, that is still a bill most people and companies cannot carry.

This post is about the clever way around it. It will turn out that to teach a giant model a new trick, you do not need to move the whole giant at all - it is enough to attach a tiny “add-on” amounting to a fraction of a percent of its weights. And why this works at all hides one of the more surprising observations about the nature of these models: that a new skill has a strikingly small size. This is the last stop of the series - and at the same time the one that best shows how much of what a model “knows” already sits inside it, before we ask it anything.

💡 How to read this. The main thread is descriptive and self-contained. I tuck the formulas into collapsible “for the curious” blocks - if you like math, expand them; if not, skip them with no loss of understanding. Every claim has a footnote to its source paper at the end.

📋 TL;DR

Full fine-tuning is expensive. Updating all the weights of a 70B model requires almost as much hardware as training from scratch. LoRA was created precisely as an answer to how unaffordable this is at GPT-3 scale.
LoRA: train only the correction. We freeze the original weights and train a tiny, “low-rank” correction written as the product of two thin matrices. At GPT-3 175B scale that is about 10,000× fewer trainable parameters and about 3× less GPU memory - at a quality on par with full fine-tuning.
No cost at inference. After training, the correction can be merged back into the weights, so the finished model runs exactly as fast as before - unlike the older “adapters”, which slowed inference down.
Why it works at all. Because adapting to a new task has a low intrinsic dimension: in one experiment a mere 200 parameters were enough to reach 90% of full fine-tuning’s score. If the task itself is “small”, then the correction can be small too.
The whole PEFT family. LoRA is one of many ways to fine-tune efficiently (adapters, prefix/prompt tuning, BitFit). All of them move a fraction of the weights, giving a result close to full fine-tuning.
QLoRA: cheaper still. We quantize the frozen model to 4 bits and train the adapters on that squeezed version - letting you fine-tune a 65B model on a single 48 GB card, with no loss of quality.
Why fine-tune. Fine-tuning is exactly what turns a raw “text-completer” model into an assistant: instruction tuning teaches it to follow commands, and RLHF to align with human preferences. Aligning with intent can beat sheer size.

👉 Play with it: the interactive page has a section on LoRA - see how a frozen matrix gets a cheap “side track” of two thin matrices, and how much that saves.

🧊 LoRA: train only the correction

Let us start with the trick itself. Picture a large, trained weight matrix $W_0$ - one of thousands in the model. Full fine-tuning would say: change all of it, every one of its millions of numbers. LoRA (Low-Rank Adaptation; Hu et al., 2021) says something else: leave $W_0$ frozen, do not touch it at all, and alongside it add a small correction $\Delta W$ (read “delta W”) that you learn¹.

All the cleverness lies in how we store that correction. Instead of keeping it as yet another full, dense matrix (just as big as $W_0$ - so no saving at all), we factor it into the product of two thin matrices: $B$ and $A$ . If the original is, say, 4096 by 4096, then instead of learning a correction of over 16 million numbers, we learn two narrow strips - one tall and thin, the other low and wide - which, multiplied together, give a matrix of the same shape, but “pinched” in the middle to just a few dimensions. That number of dimensions in the pinch is the rank, written $r$ - usually 8 or 16. It decides how many new parameters we add at all.

The numbers from the original paper are striking. Relative to full fine-tuning of GPT-3 175B with the Adam optimizer, LoRA reduces the number of trainable parameters by about 10,000 times and the GPU memory requirement by about 3 times - while reaching results on par with, or better than, full fine-tuning¹. For a comparison with the interactive page: at dimension 4096 and rank 8 this works out to about 256× fewer correction parameters.

There is one more, crucial advantage over the older methods. Since the correction is simply something we add to $W_0$ , once training is done it can be merged back into the original matrix - compute $W_0 + \Delta W$ once and store it as a single matrix. As a result the finished, fine-tuned model has exactly the same structure and the same speed as the starting model: zero added latency at inference¹. This is what sets LoRA apart from the “adapters” we will meet shortly - those added new layers that have to be computed on every query.

In practice you do not glue LoRA onto every matrix in the model. The authors noticed that, for a fixed parameter budget, the most is gained by adapting the query and value matrices in the attention mechanism (the $W_q$ and $W_v$ from the post on attention) - and that is where the default convention comes from¹. A second finding was interesting too: it is better to adapt more matrices at a lower rank than fewer matrices at a higher rank¹.

📐 For the curious: the low-rank factorization and parameter count

LoRA freezes a trained matrix $W_0 \in \mathbb{R}^{d \times k}$ and learns only a low-rank correction, factored into two matrices:

$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r}\,BA, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll \min(d,k)$

For an input $x$ the layer’s value is the sum of the (frozen) main path and the cheap side track:

$h = W_0\,x + \frac{\alpha}{r}\,B(A x)$

Here $\alpha$ (alpha) is the scale of the correction’s influence, and $r$ is the rank - the width of the pinch. Where does the saving come from? The full correction $\Delta W$ has $d \times k$ numbers; the factorization into $B$ and $A$ has $d \times r + r \times k = r(d+k)$ . For a square $d \times d$ matrix that gives $2dr$ instead of $d^2$ , a reduction by a factor of $\frac{d}{2r}$ - for $d = 4096$ and $r = 8$ about 256× fewer¹.

🔬 The low-rank hypothesis: why it works at all

Here comes a question easy to miss amid the delight over the saving: why does a low-rank correction suffice at all? After all, we are squeezing the weight change down to just a few dimensions - intuition says we must be losing something. And yet we lose almost nothing. Why?

The answer reaches back to an observation older than LoRA itself. Aghajanyan and colleagues (2020) studied something they called the intrinsic dimensionality of a task. The question was: how many “degrees of freedom” do you really need to fine-tune a large model to a new task? The result was surprising. It turned out that optimizing a mere 200 parameters - then projected randomly into the full, multimillion-dimensional weight space - was enough for the RoBERTa model to reach 90% of the score of full fine-tuning on a certain language task². Two hundred numbers. Not two hundred million.

This is the crux. If the task itself has a low intrinsic dimension - if its “shape” can be described by a handful of numbers - then the needed weight change $\Delta W$ can be low-rank too. LoRA does not blindly guess that the correction is “small”; it rests on the empirical discovery that fine-tuning a large, trained model really does live in a strikingly tight space. A large model does not learn a new task from scratch - rather it shifts by a small step within a space it has already mastered.

There is also an elegant technical detail that shows how carefully the method was designed. At the start of training we want the correction to break nothing - the model has to start exactly from its trained weights. So one of the matrices ( $A$ ) is initialized randomly, and the other ( $B$ ) with zeros. A product with zero is zero, so at the start $\Delta W = 0$ : the model begins untouched and only gradually, during learning, drifts away from the original¹.

📐 For the curious: intrinsic dimension via random projection

Measuring the intrinsic dimension works by, instead of optimizing the full parameter vector $\theta \in \mathbb{R}^{D}$ (where $D$ runs into the millions), optimizing a tiny vector $\theta^{(d)} \in \mathbb{R}^{d}$ in a space of dimension $d \ll D$ , and then projecting it back with a fixed, random matrix $P$ :

$\theta = \theta_0 + P\,\theta^{(d)}$

Here $\theta_0$ (theta zero) are the frozen starting weights, and $P \in \mathbb{R}^{D \times d}$ is random and fixed (we do not learn it). The “intrinsic dimension” is the smallest $d$ at which the model reaches a target percentage of full fine-tuning’s score. Aghajanyan’s finding: for many tasks that $d$ is on the order of hundreds, not millions². This is exactly the observation LoRA’s intuition rests on - that $\Delta W$ can be low-rank.

🧰 The whole PEFT family

LoRA did not come out of nowhere - it is the best-known member of a broader family of methods called PEFT (Parameter-Efficient Fine-Tuning). The common denominator of the whole family: instead of moving all the weights, we adapt a small part of them (or add a handful of new ones), achieving a result comparable to full fine-tuning³. It is worth knowing a few of LoRA’s cousins, because each attacks the same problem from a different angle:

Adapters (Houlsby et al., 2019) - historically the first. They insert small, trainable modules between the existing layers, freezing the rest. On the GLUE benchmark they came within 0.4% of full fine-tuning, adding only 3.6% of the parameters per task⁴. The drawback that LoRA later eliminated: those extra modules are real layers that must be computed on every query, so they slow the model down at inference¹.
Prefix tuning (Li and Liang, 2021) - instead of moving the weights, it prepends a short, learned sequence of “virtual tokens” that steers the model’s behavior. It trains a mere 0.1% of the parameters⁵.
Prompt tuning (Lester et al., 2021) - a related idea: it learns “soft prompts” with the model fully frozen. An interesting observation: the larger the model, the more this lightweight method closes the gap to full fine-tuning⁶.
BitFit (Ben-Zaken et al., 2021) - perhaps the most minimalist: it modifies only the bias terms (those constant offsets each neuron adds to its sum). Even so it can be competitive with full fine-tuning for smaller datasets⁷.

What unites all these methods is the same deep observation from the previous section: a large model is already so richly “shaped” by pretraining that a new task needs only a gentle nudge. The official HuggingFace PEFT library today collects more than a dozen such methods under a single interface - LoRA being the default starting point among them³.

🗜️ QLoRA and quantization: lower still

LoRA radically cut the cost of training the correction. But there is a second cost that does not vanish: to fine-tune anything, the frozen base model still has to be loaded into the graphics card’s memory. And a 65B model in ordinary 16-bit precision is well over a hundred gigabytes - more than a single card holds. This is where quantization comes in.

Quantization is simply lowering the precision of the numbers that store the weights. Instead of keeping each weight in 16 bits, we pack it into 8, or even 4 bits - like saving the same photograph at a lower resolution. Fewer bits means less memory and faster computation, at the cost of some loss of accuracy. The art is in minimizing that loss. Two important papers showed it can be done almost painlessly: LLM.int8() (Dettmers et al., 2022) compressed the weights to 8 bits without loss of quality, cleverly handling the few “outlier” values that would wreck naive quantization⁸. GPTQ (Frantar et al., 2022) went to 3-4 bits, quantizing a 175B model in a few hours and speeding up its inference more than threefold⁹.

QLoRA (Dettmers et al., 2023) brilliantly combined both worlds: quantization and LoRA. The idea - freeze the base model and compress it all the way down to 4 bits, then train LoRA adapters on it, passing the gradients through that 4-bit version¹⁰. The base giant lies tightly packed in memory, while only the thin add-on is trained. The result? A 65B model can be fine-tuned on a single 48 GB card - while preserving the quality of full, 16-bit fine-tuning¹⁰. For scale: the Guanaco family of models trained this way reached 99.3% of ChatGPT’s level after just 24 hours of fine-tuning on a single GPU¹⁰. Fine-tuning large models stopped being the exclusive domain of large labs.

📐 For the curious: how to pack a weight into 4 bits

Linear quantization maps floating-point numbers onto a small, discrete grid of integers. In simplified terms, for a block of weights we compute a scale factor $s$ from the largest absolute weight in the block, and then:

$w_{\text{int}} = \mathrm{round}\!\left(\frac{w}{s}\right), \qquad w \approx s \cdot w_{\text{int}}$

where $w_{\text{int}}$ fits in the range allowed for the given number of bits (for 4 bits that is a mere 16 levels). The factor $s$ plays the role of the “grid step”. QLoRA adds two tricks on top: the NF4 data type (4-bit NormalFloat), whose levels are laid out optimally for weights with a near-normal distribution, and double quantization - it quantizes the scale factors themselves, saving about another 0.37 bit per parameter¹⁰. During training the gradients flow through this 4-bit path to the LoRA adapters, which stay at higher precision.

🧬 After LoRA: the family’s variants

LoRA proved so successful that it quickly grew improvements. You do not need to know them to use LoRA - but it is worth knowing this is a living, evolving family, not a closed recipe:

DoRA (Liu et al., 2024) - decomposes each weight into a magnitude and a direction, applying LoRA only to the direction. Improves learning capacity with no added inference cost¹¹.
LoRA+ (Hayou et al., 2024) - notices that the two correction matrices ( $A$ and $B$ ) should learn at different speeds. A small change, yet it brings an improvement and up to a two-fold training speedup¹².
AdaLoRA (Zhang et al., 2023) - instead of giving every matrix the same rank, it allocates the budget of ranks where they help most, pruning the unimportant directions¹³.
rsLoRA (Kalajdzievski, 2023) - returns to a small but consequential detail: how to scale the correction. It proves that with the right scaling factor, higher ranks finally start to help the way they should¹⁴.

📐 For the curious: the scaling factor in rsLoRA

The original LoRA scales the correction by $\frac{\alpha}{r}$ (alpha over r). rsLoRA argues that this scaling “chokes” the gradients at larger ranks, and proposes dividing instead by the square root of the rank:

$\Delta W = \frac{\alpha}{\sqrt{r}}\,BA$

The intuition is the same as behind the $\sqrt{d_k}$ factor in the attention mechanism: the square root stabilizes the variance as we increase the number of summed terms. With the right factor, higher ranks stop hurting and start to give better fine-tuning¹⁴.

🧭 When to use what: fine-tuning, RAG, or just a prompt

Before reaching for LoRA, it is worth stepping back and asking whether you need fine-tuning at all. Because it is only one of four tools for “bending” a model to your needs - and often not the right one. There is no single canonical comparative paper here; below is a practical synthesis:

Just a prompt (prompting / few-shot) - no training at all. If the task can be described with an instruction or shown with a few examples in the query itself - start here. Cheapest and instant, limited only by the size of the context window.
RAG (adding knowledge from retrieval) - when you need current or in-house factual knowledge. RAG adds the relevant documents to the model’s context instead of changing its weights. It teaches no new behavior or style - it provides facts.
PEFT (LoRA/QLoRA/…) - when you need to change the behavior, style, or format of the answers, or adapt the model to a specific domain, with limited hardware and many tasks (lightweight, swappable adapters)¹¹⁰.
Full fine-tuning - maximum learning capacity, when you have large resources and lots of data. For 70B+ models, expensive - and it was that cost that called LoRA into being¹.

A rule of thumb worth remembering: missing knowledge → RAG; missing behavior → fine-tuning. Confusing the two is the most common mistake - fine-tuning a model so it “knows facts” is usually more expensive and worse than simply giving it those facts in context.

🎓 Why fine-tune at all: from a raw model to an assistant

There is one more use of fine-tuning - the most important of all, because without it there would be no ChatGPT or Claude as you know it. The raw model after pretraining, the one we assembled in the previous posts, can do only one thing: complete text, predict the next token. Asked “What is the capital of Poland?” it might just as happily go on with “And what is the capital of France?” - because statistically that is what texts with questions look like. It does not know it is supposed to answer. It is fine-tuning that turns such a “completer” into an assistant.

This happens in two steps. The first is instruction tuning - fine-tuning on a set of commands and their correct executions. The FLAN paper (Wei et al., 2021) showed something surprising: a model fine-tuned on over 60 different tasks phrased as instructions suddenly gets better at tasks it has never seen. The very form “this is a command, carry it out” turned out to be a skill that generalizes¹⁵.

The second step is even more interesting: RLHF (Reinforcement Learning from Human Feedback). Here we no longer show the model “correct answers”, because for many questions there is no single correct one - we want an answer that is helpful, safe, in good tone. So instead we show it human judgments: which of two answers a person preferred. From those preferences a reward model is trained, and then the main model is fine-tuned to maximize that reward. The result can be striking: in the InstructGPT paper (Ouyang et al., 2022) a 1.3-billion model fine-tuned with RLHF was preferred by humans over the raw 175-billion model - more than a hundred times larger¹⁶. Aligning with human intent beat sheer size.

And here it all comes together: LoRA and QLoRA are, in practice, a frequent mechanism for carrying out instruction tuning and RLHF on limited hardware. The Guanaco mentioned above is exactly such a case - an assistant fine-tuned with QLoRA¹⁰. So the clever saving from the start of this post turns out to be not an engineering curiosity, but the thing that democratizes the building of assistants: that last, most important layer of “being helpful” can today be added at a small cost.

📐 For the curious: the optimization objective in RLHF

In RLHF a reward model $r_\phi$ is trained first, which from human preferences (winning vs losing answer) learns to assign texts a numeric “quality”. Then the main model $\pi_\theta$ is fine-tuned to maximize the expected reward, but without drifting too far from the starting model $\pi_{\text{ref}}$ :

$\max_{\theta}\ \mathbb{E}\big[\,r_\phi(x, y)\,\big] - \beta\,\mathrm{KL}\!\big(\pi_\theta \,\|\, \pi_{\text{ref}}\big)$

The second term is a penalty for divergence (the Kullback-Leibler divergence) between the fine-tuned and the starting model, weighted by a coefficient $\beta$ (beta). Without it the model would “overshoot” toward the reward model, generating odd texts that artificially rack up points (so-called reward hacking). The KL penalty keeps it close to the sensible language it learned in pretraining¹⁶.

🌗 The knowledge that was already there

Let us pause at the end, because this whole story about saving hides something stranger than it first appears. On the surface LoRA is just clever engineering - a trick that saves memory. But recall the number from the low-rank hypothesis: two hundred parameters were enough for a giant model to learn a new task to 90%. Not two hundred million. Two hundred.

Think about what that actually means. In the second post of the series we marveled at billions of parameters, the colossal cost of training, the trillions of tokens. And now it turns out that to re-aim that giant at a new task, it is enough to move a handful of numbers. How is that possible? The simplest explanation is also the deepest: because the new skill is not really being added. It is, in some form, already there - latent in the weights that absorbed almost the entire written record of humankind. Fine-tuning does not so much teach the model from scratch as find and bring forward what pretraining already arranged inside it. A small add-on suffices because it builds no new ability - it only turns the spotlight toward one that was waiting in the dark.

This is perhaps the quietest, yet deepest moral of the whole series. We took the LLM apart piece by piece - neuron, network, token, attention, block - looking for where what the model can do resides. And LoRA shows that this ability is not localized in any one place that could be fine-tuned separately; it is smeared across that whole trained geometry so densely that a new behavior can be drawn out of it with the barest touch. The anatomy is coming to a close - we have taken the mechanism apart down to the last screw. And the question of how so many latent abilities fit into that frozen tissue, that a gentle nudge surfaces yet another, is already a question that reaches beyond the blueprint itself. I leave it to a separate essay on the human-LLM resonance, because that is its proper place.

🎯 What’s next: the end of the road from neuron to LoRA

That was the last stop. We began with a single neuron - a threshold, a weighted sum, an activation function - and arrived all the way here: at a trick that lets you take a finished, multimillion-dollar giant and specialize it on your own graphics card, moving a fraction of a percent of its weights. Along the way we assembled networks and training, tokens and embeddings, the attention mechanism and the whole Transformer block. Seven stops, one mechanism - taken apart down to the last screw.

And here the loop closes back to the overview post where this journey began. We learned how it is built - from strikingly simple, repetitive blocks. But the more closely we took the machine apart, the more clearly it showed that why something that reasons, translates and holds a conversation grows out of that simplicity remains open. The anatomy did not take the mystery away - it only showed exactly where it lives.

If you would rather touch it now than read on, the interactive “Anatomy of an LLM” page lets you play with each of these blocks - from the neuron to LoRA. And if you are drawn to what actually emerges from this machinery - and why a conversation with a model can resemble a resonance with another mind - I wrote about that separately. That is where this series finds its closing word.

E. J. Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models (freezing $W_0$ , low-rank correction $\frac{\alpha}{r}BA$ , ~10,000× fewer parameters and ~3× less memory for GPT-3, no inference latency via merging, the $W_q$ / $W_v$ convention). arXiv:2106.09685. https://arxiv.org/abs/2106.09685 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸ ↩⁹ ↩¹⁰
A. Aghajanyan, L. Zettlemoyer, S. Gupta (2020). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning (low intrinsic dimension of tasks; 200 parameters → 90% of the score on MRPC for RoBERTa). arXiv:2012.13255. https://arxiv.org/abs/2012.13255 ↩ ↩²
HuggingFace PEFT - documentation (Parameter-Efficient Fine-Tuning; methods fine-tune a small fraction of parameters at quality close to full fine-tuning; the set of methods includes LoRA, AdaLoRA, DoRA, prefix/prompt/p-tuning, among others). https://huggingface.co/docs/peft/index ↩ ↩²
N. Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP (adapters; within 0.4% of full fine-tuning on GLUE at 3.6% of the parameters per task). ICML. arXiv:1902.00751. https://arxiv.org/abs/1902.00751 ↩
X. L. Li, P. Liang (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation (a learned continuous prefix, ~0.1% of parameters). ACL. arXiv:2101.00190. https://arxiv.org/abs/2101.00190 ↩
B. Lester, R. Al-Rfou, N. Constant (2021). The Power of Scale for Parameter-Efficient Prompt Tuning (soft prompts with a frozen model; larger models close the gap). EMNLP. arXiv:2104.08691. https://arxiv.org/abs/2104.08691 ↩
E. Ben-Zaken, S. Ravfogel, Y. Goldberg (2021). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (modifying only the bias terms). arXiv:2106.10199. https://arxiv.org/abs/2106.10199 ↩
T. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (INT8 quantization without loss of quality, handling “emergent outlier features”). NeurIPS. arXiv:2208.07339. https://arxiv.org/abs/2208.07339 ↩
E. Frantar et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (quantizing a 175B model to 3-4 bits in ~4 GPU hours, ~3.25× inference speedup). ICLR. arXiv:2210.17323. https://arxiv.org/abs/2210.17323 ↩
T. Dettmers et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs (4-bit NormalFloat, double quantization, paged optimizers; fine-tuning 65B on a single 48 GB GPU; Guanaco at 99.3% of ChatGPT’s level in 24 h). NeurIPS. arXiv:2305.14314. https://arxiv.org/abs/2305.14314 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶
S.-Y. Liu et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation (decomposing a weight into magnitude and direction; LoRA only on the direction). ICML. arXiv:2402.09353. https://arxiv.org/abs/2402.09353 ↩
S. Hayou, N. Ghosh, B. Yu (2024). LoRA+: Efficient Low Rank Adaptation of Large Models (different learning rates for $A$ and $B$ ; 1-2% improvement and up to 2× speedup). ICML. arXiv:2402.12354. https://arxiv.org/abs/2402.12354 ↩
Q. Zhang et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (adaptive allocation of the rank budget by importance, an SVD-form parameterization). ICLR. arXiv:2303.10512. https://arxiv.org/abs/2303.10512 ↩
D. Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (scaling by $\frac{\alpha}{\sqrt{r}}$ instead of $\frac{\alpha}{r}$ ; with the right factor higher ranks help). arXiv:2312.03732. https://arxiv.org/abs/2312.03732 ↩ ↩²
J. Wei et al. (2021). Finetuned Language Models Are Zero-Shot Learners (FLAN; instruction tuning on 60+ tasks improves zero-shot; outperforms GPT-3 175B on 20 of 25 tasks). ICLR. arXiv:2109.01652. https://arxiv.org/abs/2109.01652 ↩
L. Ouyang et al. (2022). Training language models to follow instructions with human feedback (InstructGPT; SFT + RLHF; a 1.3B model preferred over GPT-3 175B; the objective with a KL penalty). NeurIPS. arXiv:2203.02155. https://arxiv.org/abs/2203.02155 ↩ ↩²