Bigger batch size is not always faster: fine-tuning Gemma 3 270M on four GPUs
I recently fine-tuned Google’s Gemma 3 270M following Daniel Bourke’s Hugging Face full fine-tuning tutorial, and I got curious about one thing: how much does the GPU, and the batch size I pick, change how long training takes? So I ran the same job on four different GPUs, changed the batch size from run to run, and logged everything to Weights & Biases.
The result surprised me. On two of my GPUs, a bigger batch size made training roughly 4x to 10x slower, not faster. This post walks through what I ran, what I saw, and why it happens.
The setup
Model: google/gemma-3–270m-it (about 268M parameters), loaded in bfloat16 with eager attention.
Data: the mrdbourke/FoodExtract-1k dataset, converted to a prompt/completion chat format, with an 80/20 train/test split. That works out to roughly 1,100 training examples.
Training: Hugging Face TRL’s SFTTrainer, the same config on every machine:
- 3 epochs, learning rate 5e-5, constant schedule
- adamw_torch_fused optimizer
- max sequence length 512, no packing
- completion-only loss, gradient checkpointing off (except for two runs, see section 5)
- evaluation and checkpointing once per epoch
The only thing I changed between runs was per_device_train_batch_size (and the evaluation batch size with it), plus the hardware, and gradient checkpointing in two runs (section 5).
Hardware:
- RTX 5090 Laptop: about 24 GB VRAM, Windows 11
- RTX 4050 Laptop: 6 GB VRAM, Windows 11
- GTX 1060 Laptop: 6 GB VRAM, WSL2 on Windows
- Tesla T4: 15 GB VRAM, Google Colab
Every GPU except the T4 was a laptop part, so these numbers are not the same as a desktop card with the same name.
Results
These are the training times for completed 3-epoch runs, taken from the trainer’s own training runtime. Lower is better. Times are written as hours, min and sec. Each line shows batch size, training time, samples per second and peak VRAM.
RTX 5090 Laptop
- Batch size 10: 1 min 30 sec, 38.0 samples/sec, 11.4 GB
- Batch size 16: 1 min 29 sec, 38.4 samples/sec, 15.6 GB
- Batch size 22: 1 min 32 sec, 37.2 samples/sec, 18.9 GB
- Batch size 32: 6 min 28 sec, 8.8 samples/sec, 23.5 GB
RTX 4050 Laptop
- Batch size 4: 6 min 13 sec, 9.1 samples/sec, 5.8 GB
- Batch size 10: 1 hour 3 min 44 sec, 0.89 samples/sec, 5.8 GB
Tesla T4 (Colab)
- Batch size 4: 10 min 24 sec, 5.4 samples/sec, 6.0 GB
GTX 1060 (WSL2)
- Batch size 4: 1 hour 42 min 59 sec, 0.55 samples/sec, 5.9 GB
Accuracy was the same across all of these runs by the end, so the comparison here is purely about time.
What I found
1. Past a point, a bigger batch buys nothing
On the 5090, going from batch size 10 to 16 to 22 changed training time by only a few seconds. All three finished in about 1 min 30 sec. A 270M model with sequences capped at 512 tokens simply does not have enough work per step to keep making use of a bigger batch, so throughput flatlines at around 38 samples per second.
2. Then there is a cliff
At batch size 32 the 5090 took 6 min 28 sec, about 4.2x longer than at batch size 22, even though it had been running smoothly one step earlier. Peak VRAM at that point was 23.5 GB on a card with about 24 GB.
Get Qureshimuhammadabubakar’s stories in your inbox
Join Medium for free to get updates from this writer.
The 4050 is the extreme case. At batch size 4 it trained in 6 min 13 sec. At batch size 10 it took 1 hour 3 min 44 sec, about 10x slower, and the card was already at 5.8 GB of its 6 GB in both runs.
3. Why it slows down: memory spilling into system RAM
The slowdown comes from running out of VRAM. When a GPU’s dedicated memory is full, NVIDIA’s Windows drivers (since version 536.40) can spill allocations into shared GPU memory, which is system RAM reached over PCIe, instead of failing with an out-of-memory error (NVIDIA). That memory is far slower than VRAM, and PyTorch does not tell the two apart, so training simply crawls (PyTorch forum).
The 4050 laptop shows it clearly. Batch size 10 needs more than its 6 GB, and the overflow lands in its roughly 16 GB of RAM: the machine used up to 67% in the fast run (batch size 4) and 82% to 94% in the slow one, peaking at 14.4 GB used. The training process alone reached 6.6 GB against 2.6 GB, even though VRAM was about 96% full in both runs.
The 5090 laptop does the same at batch size 32, which fills 23.5 GB of its roughly 24 GB. With about 64 GB of RAM to spare, the spill barely shows in the RAM numbers (the process used 3.9 GB against 2.7 GB, and the machine 53% against 50% to 51%), but training is still 4.2x slower than at batch size 22.
The GTX 1060 is a different case. WSL2 was given 16 GB of the PC’s 32 GB, and the run peaked at about 5.6 GB of it (36%), so RAM was not the bottleneck there. Its slowness comes from the card itself (see section 4).
4. GPU generation matters a lot
Comparing each GPU at its best completed run:
- The 5090 laptop (about 1 min 29 sec) was about 4x faster than the 4050 laptop (6 min 13 sec).
- It was about 7x faster than the T4 (10 min 24 sec).
- It was roughly 70x faster than the GTX 1060 (1 hour 42 min 59 sec).
The 1060 is a Pascal card with no native bfloat16 support, which very likely explains a good part of that gap, since the whole job was run in bf16.
One note on the T4: the training itself took 10 min 24 sec, and that is the number to trust. The Colab run kept showing as running in Weights & Biases after the notebook cells had finished, so its wall-clock time (45 min 55 sec) is stretched by idle time, which likely also explains its low average GPU utilization (about 19%).
5. Gradient checkpointing trades speed for memory
Two more runs, one on the GTX 1060 and one on the T4, had gradient checkpointing turned on. It saves memory by recomputing activations in the backward pass instead of storing them. I kept these two out of the main results because they are not a like-for-like comparison, but the pairs below show what the trade looks like on the older cards. Each line shows batch size, checkpointing, training time, samples per second and peak VRAM.
GTX 1060 (WSL2)
- Batch size 4, checkpointing off: 1 hour 42 min 59 sec, 0.55 samples/sec, 5.9 GB
- Batch size 10, checkpointing on: 1 hour 52 min 53 sec, 0.50 samples/sec, 5.8 GB
Tesla T4 (Colab)
- Batch size 4, checkpointing off: 10 min 24 sec, 5.4 samples/sec, 6.0 GB
- Batch size 14, checkpointing on: 18 min 45 sec (stopped at 2.1 of 3 epochs), 3.0 samples/sec, 4.0 GB
On the GTX 1060, batch size 10 with checkpointing was about 8% slower per sample than batch size 4 without it, and the 6 GB card was full either way. The bigger batch bought nothing.
On the T4, batch size 14 with checkpointing was about 44% slower per sample, but it peaked at 4.0 GB against 6.0 GB, even with a batch 3.5x larger. So checkpointing is a way to fit a bigger batch, not a way to go faster, and on these two cards the bigger batch did not make up for the extra compute.
The T4 checkpointing run stopped at about 2.1 of 3 epochs, so only its samples per second are comparable, not its total time.
Takeaways
- Don’t assume bigger is faster. Throughput can plateau well before you run out of memory, and past that point you are only using more VRAM.
- Stay comfortably under your VRAM limit. The two big slowdowns both happened when peak VRAM got within a hair of the card’s capacity.
- Try a few batch sizes on your own hardware. It took me only a handful of short runs to find the sweet spot, and the difference between the best and worst choice on the same card was 4x to 10x.
- Old or small GPUs can still do this job, just slowly. A GTX 1060 still fine-tuned the model, in 1 hour 42 min 59 sec.
Caveats
- Each configuration ran once, so there is no error bar on any of these numbers.
- The machines differ in more than the GPU: operating system, CPU, and Python and library versions were not identical.
- Peak VRAM is the amount PyTorch reserved, which can be higher than what the model strictly needed.
- I only measured training time. Accuracy ended up the same across runs, but I did not study how batch size interacts with quality at a fixed learning rate.
Originally published at bakarqureshi.dev.
