LoRA Fine-Tuning Made Simple: Concepts and Math Explained
Understand how LoRA works, why it needs fewer trainable parameters, and how lightweight adapters let one LLM handle multiple tasks.
Introduction
A pretrained model already understands language and can chat, summarize and answer questions out of the box. It is a capable generalist. Many projects, however, need a specialist: for example, a model that writes Python in your team’s style or labels financial headlines. Training the model further on examples of that specific behavior is called fine-tuning.
There are two broad approaches to fine-tuning: Full Fine-Tuning and Parameter-Efficient Fine-Tuning (PEFT).
In full fine-tuning, every weight in the model is updated. It works well, but it is expensive because it demands large amounts of compute and memory.
PEFT is much lighter. It freezes the original weights and trains only small additional weight matrices, often called adapters. Training needs far fewer resources, the output is a file of just a few megabytes, and one base model can be reused with a different adapter for each task.
LoRA (Low-Rank Adaptation) is the most widely used method in the PEFT family.
In this post, we’ll see why LoRA fine-tuning is lighter and faster, then walk through hands-on code that runs one base model with multiple adapters.
Why LoRA is faster and lighter: let’s calculate
Let’s do some simple but eye-opening calculations to see why LoRA fine-tuning is so much faster and lighter. Don’t worry if you’re not familiar with the transformer architecture. I’ll keep everything simple and easy to follow.
I will use Qwen2.5–0.5B-Instruct as example model to explain the concepts. Let’s go step by step.
1) A model has a set of weights
At its core, a transformer model is a large collection of weights. These weights are grouped into layers, and every layer contains the same seven main weights:
- Attention weights:
Wq(query),Wk(key),Wv(value),Wo(output) - MLP (feed-forward) weights:
gate,up,down
The same set of seven repeats in every layer. So if you know the size of one layer, you can multiply by the number of layers to get the size of almost the whole model.
2) Each weight is a matrix
Every weight is a table of numbers, in other words a matrix. How big each matrix is depends on a few settings in the model’s configuration. For Qwen2.5–0.5B-Instruct model, these are:
- Hidden size = 896
- Intermediate size = 4864
- Key/value size = 128
If you’re new to transformers, don’t worry about what each setting means. Just think of them as the “dimensions” that decide the size of each weight matrix. With these configuration values, the weight matrices look like this:
3) Let’s count number of parameters in the model
Since each weight is a matrix (a table of numbers), we can find its number of parameters by multiplying its rows by its columns. For example, Wq is 896 × 896 = 802,816 parameters. The table below shows this count for every weight. Adding all seven together gives the number of parameters in one layer.
Qwen2.5–0.5B-Instruct model stacks 24 layers, and each layer has its own copy of these seven weights. So the total is simply:
Total parameters = parameters in one layer × number of layers
= 14,909,440 × 24 = 357,826,560 (~358 million)
This means full fine-tuning has to update about 358 million parameters across these layers. Keep this number in mind, because we’ll come back to it soon.
4) The LoRA trains less number of parameters
LoRA works differently. Instead of changing the original weights, it trains small adapters. Every base weight gets its own matching adapter, and during training the base weights stay frozen while only the adapters are trained.
Each adapter is made of two small matrices, A and B. Their size depends on a setting called the rank (r), which controls how “wide” the adapter is. A bigger r means a larger adapter that can learn more, but it also costs more to train. You choose r when you set up LoRA fine-tuning. Common values are 8, 16 or 32.
Get Brajendra Singh’s stories in your inbox
Join Medium for free to get updates from this writer.
Let’s use r = 16. For each of the seven base weights we saw earlier, the A and B matrices look like this:
Multiplying rows by columns again gives the number of parameters in A and B. Adding them up for all seven adapters gives 366,592 trainable parameters per layer.
Now multiply by the 24 layers:
Trainable parameters = 366,592 × 24 = 8,798,208 (~8.8 million)
So full fine-tuning has to train 358 million parameters, while LoRA trains only 8.8 million. That’s about 40× fewer. In practice, that means less GPU memory, faster training steps, and often one small GPU instead of several large ones.
So if LoRA is so efficient then why not use it always?
What’s the catch?
LoRA trains far fewer parameters, so it has less capacity to learn. It is excellent for teaching a model a new style, format or narrow task, but when a model must learn a lot of new knowledge (a new language or a large specialist domain), full fine-tuning usually performs better. A good rule of thumb: start with LoRA, and move to full fine-tuning only if the results aren’t good enough.
Once LoRA weights are trained, how do we use it?
Using the trained LoRA weights
There are two ways a base model can use the trained LoRA weights:
Option A: Merge the adapter into the base model The adapter’s changes are added directly into the base weights and the result is saved as a normal model.
- ✅ Pros: No extra latency at inference. It behaves exactly like a regular model.
- ❌ Cons: You need one full model per task. Switching tasks means loading a different full model.
Option B: Keep the adapter separate and load it with the base model The base model is loaded once, and adapters are attached on top of it.
- ✅ Pros: One base model in memory, plus only a few MB per adapter. You can switch adapters at runtime depending on the adapter specialist a request needs.
- ❌ Cons: A small extra compute and latency cost at inference.
Option B is the pattern behind multi-LoRA serving in tools like vLLM and SageMaker inference components, where one endpoint serves many fine-tuned behaviors.
There is not learning without running some code.
Try it yourself: one base model, two adapters
To see all of this in action, I’ve prepared a ready-to-run Jupyter notebook: https://github.com/brajens/genai-playground/blob/main/experiments/LoRA/lora_multi_adapter_inference.ipynb
It runs on a SageMaker notebook or on your own laptop. It uses one base model and two LoRA adapters, each fine-tuned for a different task:
Here’s what the notebook does:
- Loads both adapters onto the same base model. The base model sits in memory only once, and each adapter adds just a few MB.
- Switches between the adapters instantly. There’s no reloading, and the base weights are never touched.
- Compares the results side by side: base model vs. python adapter vs. finance adapter, for every prompt.
Note: both adapters are community uploads from the Hugging Face Hub.
Conclusion
Full fine-tuning rewrites every weight of a model. For Qwen2.5–0.5B that means hundreds of millions of parameters, plus their gradients and optimizer state. LoRA takes a smarter route: it freezes the base model and trains a pair of thin matrices next to each weight. With r = 16, that’s about 8.8 M trainable parameters instead of ~358M.
The trained adapter is small and flexible. You can merge it for a fast, single-purpose model, or keep it separate and plug different adapters into one shared base model for different use cases.
Start with the notebook, watch the same base model change behavior as you switch adapters, and then train your own adapter. That’s where LoRA really clicks.
Hope you find this post useful.
