Understanding LoRA: From the Math to Fine-Tuning LLMs Better

What happens when we stop treating LoRA as a few hyperparameters and start understanding what the model is actually learning?

A few days ago, I was reading the research paper LoRA: Low-Rank Adaptation of Large Language Models.

I had seen LoRA being used in fine-tuning pipelines before. The implementation seemed pretty straightforward. You load a pretrained model, define a LoRA configuration, pass it to a trainer, and start training.

Something like this:

LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"]
)

And that’s it. We’ve configured LoRA.

But while reading the paper, I started wondering:

Why rank 16? What exactly does alpha 32 mean? And why are we training only the query and value projections?

More importantly, if our fine-tuned model doesn’t perform well, how do we know which of these settings to change?

We can always copy configurations from tutorials. But I wanted to understand what happens underneath, so that when I fine-tune a model, I’m making decisions rather than just trying different numbers.

That curiosity led me from simple matrix multiplication to understanding how LoRA learns and how this knowledge can help us train models more effectively.

Let’s start from the beginning.

1. Why do we need LoRA in the first place?

Imagine we’re building an AI system for an online shopping platform.

Every day, customers send hundreds of messages. Some want refunds, others have delivery problems, and some are facing payment-related issues.

We want our model to read these messages and classify them.

For example, a customer writes:

I ordered a pair of sneakers last week, but they still haven’t arrived.

We expect:

{
  "intent": "delivery_issue",
  "priority": "medium",
  "requires_human": false
}

Now, a pretrained LLM probably already understands what a delayed delivery means.

So why would we need to fine-tune it?

Because understanding a message and consistently following our company’s classification rules are two different things.

Maybe we have specific intent categories, escalation policies, or a required JSON format.

We could start with prompting. But suppose prompting alone isn’t giving us the consistency we need, and we have a large dataset of correctly labeled customer messages.

Fine-tuning becomes a reasonable option.

In traditional full fine-tuning, we update the pretrained model’s parameters using our dataset.

Sounds simple enough.

Until we think about how many parameters we’re updating.

Consider just one transformer weight matrix with dimensions 4,096 × 4,096.

4,096 × 4,096 = 16,777,216 parameters

That’s around 16.8 million parameters in a single matrix.

A large language model contains many such matrices across multiple transformer layers.

And training requires memory not only for model weights but also for gradients, optimizer states, and intermediate computations.

This can become expensive very quickly.

Which brings us to a question:

If our model already understands language and customer complaints, do we really need to update billions of parameters just to teach it our classification rules?

This is where LoRA comes in.

2. What if we could train just a small update instead?

Suppose we take our pretrained weight matrix, W, and freeze it.

By freezing, we simply mean its values will not change during training.

But we still need our model to learn something.

So instead of modifying W directly, what if we learn an additional matrix that represents the change we want?

Let’s call this update ΔW (delta W).

Our new weights become:

W’ = W + ΔW

In simple words:

New weights = Original weights + Learned update

This sounds promising.

But there’s a catch.

If ΔW has the same dimensions as W, we’re still training 16.8 million parameters for this matrix.

We haven’t really solved the problem.

So here’s the clever part.

What if we could represent this large update using two much smaller matrices?

That’s the central idea behind LoRA.

Instead of learning ΔW directly, we represent it as the product of two matrices, B and A.

ΔW = B × A

Our equation becomes:

W’ = W + (B × A)

We keep W frozen and train only A and B.

But how can multiplying two small matrices produce a meaningful update to a much larger matrix?

To understand that, we need to understand what low rank means.

3. How can two small matrices change a large weight matrix?

Let’s forget about billions of parameters for a moment.

Imagine we have a weight matrix with four rows and four columns.

Original weight matrix W (4 × 4)

[ w11  w12  w13  w14 ]
[ w21  w22  w23  w24 ]
[ w31  w32  w33  w34 ]
[ w41  w42  w43  w44 ]

There are 16 entries.

In full fine-tuning, each entry can be updated independently.

But LoRA takes a different approach.

Suppose we choose rank 1.

We create two matrices:

Matrix B (4 × 1)
[ 1 ]
[ 2 ]
[ 3 ]
[ 4 ]

Matrix A (1 × 4)
[ 0.1  0.2  0.3  0.4 ]

Now multiply them.

B × A =
[ 0.1  0.2  0.3  0.4 ]
[ 0.2  0.4  0.6  0.8 ]
[ 0.3  0.6  0.9  1.2 ]
[ 0.4  0.8  1.2  1.6 ]

Wait.

We started with two small matrices containing eight parameters, but their product contains 16 entries.

So we can potentially change every entry of the original weight matrix using fewer trainable parameters.

But notice something.

The second row is twice the first. The third is three times the first. The fourth is four times the first.

These changes are not independent.

That’s what the rank constraint does.

With rank 1, our update can express only a limited family of patterns. With rank 2, we can combine up to two rank-1 patterns. Higher ranks allow more complex updates.

LoRA doesn’t restrict which entries of the weight matrix can change. It restricts how independently those changes can vary.

And that’s an important distinction.

The idea is that useful task-specific changes can often be represented in a relatively low-dimensional space, without needing to update every pretrained parameter independently.

Now that we understand this, the first LoRA hyperparameter makes much more sense.

The rank.

So how do we decide what rank to use?

Let’s return to our 4,096 × 4,096 matrix.

In full fine-tuning, we would train 16,777,216 parameters.

Now suppose we choose rank 16.

Our smaller matrices become:

Matrix B             Matrix A

4096 × 16      ×      16 × 4096
               ↓
         Update matrix
           4096 × 4096

But how many parameters are we actually training?

Parameters in B = 4,096 × 16 = 65,536

Parameters in A = 16 × 4,096 = 65,536

Total = 131,072 trainable parameters

That’s approximately 131,000 instead of 16.8 million, a reduction of about 99.2% for this projection.

Now let’s compare different ranks.

These numbers apply to our example projection, not the whole model.

A higher rank gives the adapter more capacity to represent complex updates, but it also increases training cost.

So why don’t we always choose rank 64?

Because more capacity doesn’t necessarily mean better performance.

Imagine our customer-support model only needs to classify messages into a few well-defined categories. Perhaps rank 8 is already sufficient.

Increasing it to 64 might provide little improvement while requiring more parameters and computation.

On the other hand, if the task is more complex and the adapter is underfitting, increasing rank might help.

The important word is might.

We can’t know the best rank just by looking at the task.

We need experiments.

But understanding rank means we now know what we’re testing: whether our adapter needs more capacity to learn the desired behavior.

And this leads us to another question.

We’ve decided how much our adapter can learn. But where in the model should that learning happen?

4. Where should LoRA learn, and how strongly?

A transformer contains multiple weight matrices.

Inside self-attention, we have the familiar query, key, and value projections.

In simplified notation:

Query (Q) = Input × Query weights
Key   (K) = Input × Key weights
Value (V) = Input × Value weights

The model uses queries and keys to calculate attention scores, while values provide information that attention combines.

There is also an attention output projection and feed-forward layers.

So when we apply LoRA, we can choose which of these linear transformations should receive adapters.

Remember our original configuration?

target_modules=["q_proj", "v_proj"]

This tells LoRA to adapt the query and value projections, assuming the model uses these module names.

But we’re not limited to Q and V.

We could adapt all attention projections:

target_modules=[
    "q_proj",
    "k_proj",
    "v_proj",
    "o_proj"
]

Or, in supported models, adapt all linear layers:

target_modules="all-linear"

Now think about what these settings mean.

Rank determines the capacity of each adapter. Target modules determine where adaptation is allowed to happen.

Suppose our model struggles to distinguish between a refund request and a billing dispute.

We could try increasing rank while keeping the same target modules.

Or we could keep rank constant and adapt more transformer projections.

These are two different experiments.

The first tests whether the existing adapters need more capacity.

The second tests whether allowing adaptation in more parts of the network improves performance.

Neither option is guaranteed to work better, but understanding the architecture helps us make informed choices.

But what exactly does alpha do?

There’s one more important detail in the LoRA equation.

So far, we’ve been using:

W’ = W + (B × A)

But standard LoRA includes a scaling factor:

W’ = W + (alpha / rank) × (B × A)

Let’s use:

r = 16
lora_alpha = 32

The scaling factor becomes:

32 / 16 = 2

So the learned update is multiplied by 2 before being added to the original weights.

Now suppose we keep rank fixed at 16 and change alpha.

But wait.

Isn’t that what the learning rate already does?

Not exactly.

The learning rate determines how the optimizer updates the trainable parameters.

Alpha determines how the adapter’s output is scaled during the model’s computation.

Both affect optimization, but in different ways.

Get Gayatri Gattani’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Also, a scaling factor of 2 doesn’t mean the model learns exactly twice as much. The adapter matrices themselves are being optimized, so the relationship is more complicated.

There’s even a variation called rank-stabilized LoRA (rsLoRA) that uses alpha divided by the square root of rank instead of alpha divided by rank.

The idea is to improve scaling behavior, particularly when working with higher ranks.

So even a small mathematical choice in the scaling formula can affect training.

At this point, we know where LoRA is applied, how much capacity it has, and how its output is scaled.

But there’s something interesting about how training actually begins.

5. If one matrix starts at zero, how does LoRA learn?

When LoRA adapters are initialized using the standard approach, matrix A starts with nonzero random values, while matrix B starts at zero.

Remember:

W’ = W + (alpha / rank) × (B × A)

If B is zero, then B × A is also zero.

That means initially:

W’ = W

In other words, attaching the adapter doesn’t immediately change the pretrained model’s output.

That’s useful because we don’t want randomly initialized adapter weights to disturb the model before training begins.

But here’s what confused me initially.

If B starts at zero, how does the adapter learn anything?

The answer lies in how gradients flow during backpropagation.

At initialization, B is zero but A already contains random values.

This means B can receive a nonzero gradient during the first backward pass, while A initially receives a zero gradient through the adapter branch.

After the first optimizer update, B can become nonzero, allowing A to begin receiving gradients too.

So the adapter starts with no effect on the model but can still begin learning.

This is useful when debugging.

If we inspect training gradients and notice that A has a zero gradient during the very first backward pass, it doesn’t necessarily mean something is broken.

It’s expected behavior under standard LoRA initialization.

Understanding details like this helps distinguish normal training behavior from actual implementation problems.

Now let’s see how everything we’ve learned fits into a real training pipeline.

6. Let’s actually fine-tune a model

We’ll return to our customer-support classifier.

Our goal is to read a customer message and predict its intent, priority, and whether human attention is required.

We’ll use Hugging Face PEFT and TRL with a small pretrained language model.

Assume we’ve prepared a labeled training dataset in which each example has a prompt and its expected JSON completion.

For example:

Input:

I was charged twice for my order.

Expected output:

{
  "intent": "billing_issue",
  "priority": "high",
  "requires_human": true
}

In practice, we’d need many examples with consistent labeling rules, as well as separate validation and test datasets.

Now let’s configure the adapter.

from peft import LoraConfig
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
    task_type="CAUSAL_LM"
)

Notice how much more meaningful this configuration is now.

• r=16: We're allowing rank-16 updates.
• lora_alpha=32: Our standard adapter scaling factor is 2.
• target_modules: We're adapting query and value projections.
• lora_dropout=0.05: We're applying a small amount of regularization during training.

The trainer setup can then look like this:

from trl import SFTConfig, SFTTrainer
model_id = "Qwen/Qwen2.5-0.5B-Instruct"
training_args = SFTConfig(
    output_dir="./support-lora",
    learning_rate=2e-4,
    num_train_epochs=3,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    max_length=512,
    completion_only_loss=True,
    report_to="none"
)
trainer = SFTTrainer(
    model=model_id,
    args=training_args,
    train_dataset=dataset,
    peft_config=lora_config
)
trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model("./support-lora/final")

Here, dataset refers to the prepared prompt-completion training dataset. The snippet illustrates the training setup rather than being a complete standalone script.

For an instruction-tuned model, we’d also ensure that training and inference use a compatible prompt or chat-template format. The exact implementation depends on the installed library versions.

And these settings are starting points, not universally optimal values.

Now suppose training finishes successfully.

How do we know whether LoRA actually improved our model?

7. Our model is trained. How do we know if it worked?

Suppose our training loss decreases over three epochs.

That sounds promising.

But does it mean our model now classifies customer messages correctly?

Not necessarily.

Training loss tells us whether the model is optimizing the training objective. It doesn’t tell us everything about how the model behaves on unseen data.

Imagine a customer writes:

My package arrived late, the item is broken, and I’ve already been charged twice.

That’s a more complicated message than a simple delivery complaint.

Will our model correctly identify the issue according to our company’s rules?

What if it outputs valid JSON but assigns the wrong priority?

We need to evaluate the behavior that actually matters.

For our application, we could measure:

We should compare the pretrained model and the fine-tuned model on the same held-out examples.

Now suppose our LoRA model still isn’t performing well.

What should we change?

This is where everything we’ve learned becomes useful.

Instead of guessing, let’s run experiments

We might begin with rank.

Keep the same target modules and compare:

Here, alpha increases with rank to keep the standard scaling ratio equal to 2.

This lets us compare different adapter capacities under a consistent scaling ratio, although changing rank still changes the optimization problem.

If rank 8 and rank 32 achieve similar validation performance, the smaller adapter might be the better choice.

If rank 32 consistently performs better, additional capacity may be useful.

Then we could investigate target modules.

Does adapting all attention projections work better than adapting only Q and V?

Or perhaps we want to understand how scaling affects optimization. We could keep rank fixed and compare different alpha values.

The important principle is to change one major factor at a time and measure the result.

But there’s another possibility.

What if the problem isn’t rank, alpha, or target modules at all?

Suppose training loss keeps decreasing, but validation accuracy gets worse.

That could indicate overfitting.

Maybe our dataset is too small, our labels are inconsistent, or we’ve trained for too long.

Increasing rank probably isn’t the first thing to try.

Or suppose training loss barely changes.

We might inspect whether the adapters are correctly attached, whether gradients are flowing, and whether the optimizer is updating trainable parameters.

If the model produces correct classifications but invalid JSON, perhaps our training examples, prompt formatting, or decoding strategy need improvement.

Not every failure is a LoRA-capacity problem.

And that’s the real advantage of understanding the underlying mechanism.

We’re no longer blindly increasing hyperparameters. We’re identifying possible causes and testing specific hypotheses.

8. If LoRA trains so few parameters, why does it still need GPU memory?

This is another misconception I had to think through.

We saw that LoRA can reduce trainable parameters by more than 99% for a particular weight matrix.

So does that mean our total GPU memory usage decreases by 99% too?

No.

Remember, LoRA freezes the pretrained model. It doesn’t remove it.

The base model still occupies memory during training.

We also need memory for activations, adapter gradients, optimizer states, and intermediate computations.

LoRA reduces the memory associated with trainable parameters, their gradients, and optimizer states, but it doesn’t eliminate the memory required for the base model.

Now suppose the base model itself is too large to fit comfortably into GPU memory.

What can we do?

That’s where QLoRA becomes useful.

LoRA vs. QLoRA

The distinction is simple:

LoRA reduces how many parameters we train. QLoRA also reduces the memory required to store the frozen base model.

QLoRA commonly stores the pretrained weights in 4-bit quantized form while training low-rank adapters.

This can make larger models accessible on more limited hardware.

But quantization comes with trade-offs, including approximation and computational overhead.

So QLoRA isn’t automatically faster or better.

The right choice depends on what is limiting our training setup.

And once we’ve trained our adapter, there’s one final practical consideration.

What happens during deployment?

During inference, the model uses the frozen pretrained weights together with the learned adapter update.

In supported configurations, we can merge that update into the original weights:

Merged weights = Original weights + Scaled adapter update

This can simplify inference because we no longer need to compute the adapter branch separately.

But suppose our e-commerce platform has multiple AI applications — customer-support classification, product description generation, and review classification.

We might want to share one pretrained base model while using different task-specific adapters.

Keeping those adapters separate can make that possible, depending on the serving framework.

The important distinction is that LoRA makes model adaptation more efficient, but it doesn’t automatically solve every inference challenge.

The base model still needs memory, and autoregressive generation still requires attention computations and KV-cache management.

Those are separate problems addressed by inference optimizations and serving systems such as vLLM.

9. So when should we actually use LoRA?

Let’s return to our customer-support application one last time.

Suppose a customer asks:

Where is my order right now?

Would LoRA solve that problem?

Not really.

The model needs access to current order information. Connecting it to an order-management system would be more useful than fine-tuning its weights.

Now suppose the model has all the necessary information but repeatedly fails to classify customer complaints according to our defined rules.

That is a stronger fine-tuning candidate.

We should still evaluate whether prompting and structured-output techniques can solve the problem first.

But if the baseline remains insufficient and we have a representative labeled dataset, LoRA becomes worth testing.

The point isn’t that LoRA is always the best solution.

It’s that understanding how it works helps us recognize when it is the right tool.

Final thoughts

When I first looked at LoRA, I saw it as a way to fine-tune large models with fewer trainable parameters.

And that’s true.

But after understanding the mathematics, I started seeing it differently.

Rank tells us how much capacity our adapter has to represent weight updates.

Target modules tell us where those updates are allowed to happen.

Alpha controls how the learned adapter contribution is scaled.

Initialization explains how learning begins without immediately changing the pretrained model’s behavior.

And understanding memory usage explains why parameter-efficient training doesn’t automatically mean proportionally lower total GPU memory.

All of these ideas eventually connect to practical decisions.

If a model is underfitting, I now have a reason to investigate rank and target modules.

If gradients aren’t behaving as expected, I know what to inspect.

If GPU memory is the bottleneck, I understand why reducing rank may not solve the problem.

And if fine-tuning isn’t improving model quality, I know to examine the data and evaluation process before assuming that a larger adapter will fix everything.

That’s what makes understanding the foundations worthwhile.

Not because we need to manually multiply matrices every time we train a model.

But because understanding what happens underneath helps us ask better questions when things don’t work as expected.

Using LoRA makes fine-tuning more accessible. Understanding LoRA makes our fine-tuning decisions more meaningful.

And for me, that’s the real takeaway from the paper.

References

1. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models: https://arxiv.org/abs/2106.09685
2. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs : https://arxiv.org/abs/2305.14314
3. Hugging Face PEFT Documentation : https://huggingface.co/docs/peft
4. Hugging Face TRL SFT Trainer : https://huggingface.co/docs/trl/main/sft_trainer
5. Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA : https://arxiv.org/abs/2312.03732
6. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180