A 120B Model Ran About 4x Faster Than a 27B One: 8 Local LLMs on a Surface Laptop Ultra

Same llama.cpp build, same commands, one Surface Laptop Ultra (Windows on Arm, RTX Spark): generation speed lined up by active parameters, not total size, and the fastest model couldn't fix a one-character config error.

This is an English adaptation of an article I originally published in Japanese on Zenn: 120B が 27B より約 4 倍速かった。Surface Laptop Ultra でローカル LLM 8 種を同じ条件で測る. The measurements and conclusions are mine; I used an AI tool to help translate and restructure the text, and I checked the numbers against the original.

Short version: on this laptop, how fast a local LLM wrote text followed how many parameters it uses per token, not how big it is, and the fastest model couldn't get a one-character fix done.

TL;DR

• I ran 8 local LLMs through the same llama.cpp build with the same commands. Generation speed (tg128) ranged from 104.8 tokens/s (Flint) down to 12.0 (Hermes 4.3 36B).
• The order did not follow total size. gpt-oss-120B (117B total, 5.1B active per token) ran at 64.2 tokens/s, about 3.9× faster than the dense Qwen3.8-27B (16.5). The order matched the active parameter count, and I think that is the main reason; but quantization was not aligned across models, so I can't attribute the whole gap to it.
• An 86.9GB version of Nemotron 3.5 Super VL ran with every layer assigned to the GPU, although nvidia-smi showed only about 82GB of GPU memory. Where the part that didn't show up was placed, I did not measure.
• Fast didn't mean capable. In a task to fix a one-character error in a config file, the fastest model (Flint) never fixed it in 12 steps. 6 of the 8 models did.
• With 4 requests in parallel, total throughput was about 1.5–2× a single request (measured on 3 models).

Scope

• What: 8 local LLMs, loaded with the same build of llama.cpp (b12110, CUDA) and run with the same llama-bench and llama-server commands.
• On: one Surface Laptop Ultra (Windows 11 Arm64, NVIDIA RTX Spark N1X, 128GB unified memory), with about 82GB of memory assigned to the GPU.
• Measured: prompt processing and generation speed (3 runs each), reading a 32,768-token prompt (1 run), peak GPU memory, and 5 short prompts plus one config-fixing task at temperature 0 (1 run each). Also 4 parallel requests on 3 models, and an 86.9GB version of one model.
• Not aligned: quantization. Each model was used in the form it is distributed in.
• Not measured: see "What I didn't measure" near the end.
• Versions and model cards are as of November 2026, when I wrote the original.

Why I did this

In my initial setup of this laptop (Japanese) and the Ollama vs. llama.cpp comparison, I had only run one model, Qwen3.6-35B-A3B. I wanted a list: how fast other models run on this PC, and how big a model fits.

So I loaded 8 models with the same llama.cpp build and measured speed with the same commands. I also had each model answer short prompts (tool calling, code, and so on) and work on a task to fix an error in a config file, all under the same conditions.

Test setup

In the initial setup, I had assigned 95.5GB to the GPU. Later, even video generation did not use up that GPU allocation (about 46GiB at most in my comparison), while the Windows side (about 31GB) was down to about 4GB free and close to its commit limit including the page file. Leaning on the page file means more disk reads and writes, which slows things down, and I was also a little worried about SSD wear. I wanted fewer page-file writes, so I reduced the GPU assignment and gave more to Windows. My thinking was that, for the way I use this machine, the GPU assignment only needs to be as large as what I actually use.

Quantization (how many bits each weight is stored in) was not aligned. I used each model in the form it is distributed in. gpt-oss-120B is distributed in MXFP4, and Nemotron 3.5 Bolt has only one official GGUF, in the 2-bit range. For the other models I picked versions around 4 bits (except Super VL, where I used the 3-bit version that fits on the GPU). The results table lists the quantization and file size for each.

How I measured

Speed came from llama-bench:

• pp512 and pp4096: prompt processing speed (reading 512 and 4,096 tokens)
• tg128 and tg512: generation speed (writing 128 and 512 tokens)
• 3 runs each, averaged. The long-context row (read 32,768 tokens, then write 128) was run once
• GPU memory use was logged with nvidia-smi every second, and I took the peak

For usability, I started llama-server with a 32,768-token context and sent the prompts below at temperature 0. For models where thinking (reasoning before answering) can be turned off, I turned it off.

The cause in the error-fixing task: in the config file (config.yaml), port is 8O80. Where the 0 (zero) should be, there is a capital letter O.

Results

Sorted by generation speed. What the columns mean:

• Speeds are in tokens/s. "Active" is the number of parameters used per token (from each model's official model card).
• "32k read" is the speed of reading a 32,768-token prompt (pp32768).
• "GPU peak" is the highest value across the speed runs, the 32k row and the server prompts. About 2GB is in use even when nothing is running.
• "Task" is how many steps it took to fix the error-fixing task. Each tool call (list files, read, write, run a command) counts as one step, up to 12. "Not fixed" means it was not fixed within 12 steps.

• All 8 got through the 32k read without timing out.
• Over the 3 runs, the standard deviation of tg128 was 0.7 or less for every model.
• 7 models (all except Flint) passed tool calling, code and JSON. Flint passed JSON, but in tool calling it called things in the wrong order, and in code its answer was cut off at the 800-token answer limit (details below).

What I found

Speed lined up by active parameters per token

The models did not line up by total parameter count. gpt-oss-120B, at 117B, generated 64.2 tokens/s, about 3.9× faster than the 27B Qwen3.8-27B (16.5). By file size it's the same story: 63.4GB vs. 16.5GB, and the bigger one is faster.

What set the order was the number of parameters active per token. The top 5 are MoE (Mixture of Experts) models: out of many "expert" weights, they use only some for each token. gpt-oss-120B uses only 5.1B of its 117B. The bottom 3 are dense models, which use all their weights every time. In order of active parameters (about 2.5B → 3B → 5.1B → 9B → 12B → 24–36B), the speeds lined up the same way.

Each time a model writes one token, the weights it uses have to be read from memory. As far as I've read, generation speed is said to depend largely on how much has to be read. These 8 models lined up the way that explanation predicts. But quantization isn't aligned, so I can't say the whole 4× gap comes from the difference in active parameters.

From the point of view of picking a model, on this PC whether it fits on the GPU depends on file size, and how fast it runs depends on active parameters. The 63GB gpt-oss-120B is faster than the 17GB Qwen3.8-27B.

An ~87GB model still ran with everything assigned to the GPU

The 4-bit version of Super VL (UD-Q4_K_XL) is 86.9GB as a file. That's bigger than the GPU memory shown (about 82GB), so I expected it either to fail to load or to spill over to the CPU side and slow down. To find the boundary, I ran llama-bench with everything assigned to the GPU (-ngl 999).

With everything assigned to the GPU, it ran without errors, only about 10% slower than the 3-bit version (65.8GB, 26.6). Moving expert weights to the CPU side made it slower the more I moved.

nvidia-smi's usage topped out at 81,869MiB. Meanwhile, at startup llama.cpp reported 112,350MiB of GPU memory visible to CUDA (the total PyTorch saw was the same value). That's about 28GB more than nvidia-smi's 83,968MiB.

In the initial setup article, I read the total CUDA reports as close to "dedicated" plus "shared" GPU memory. If that holds here too, the part that didn't show up in nvidia-smi was placed in shared memory. But I didn't measure where it was placed. I also don't know whether the roughly 10% speed difference comes from that placement.

This test was one run per setting, and only reading 512 tokens and writing 128. I did not run the server prompts or the 32k context on it.

The 2-bit Bolt is a mixed quantization with only the experts at 2 bits

Nemotron 3.5 Bolt 74B-A9B fits a 74B model into 22GB. llama.cpp shows "IQ2_XXS - 2.0625 bpw", but that's the main type for the file as a whole. According to the official model card, the expert weights are 2-bit (IQ2-XXS), the Mamba layers are NVFP4, and the embedding and output layers are Q8_0. The card also says accuracy was recovered with quantization-aware distillation (QAD). In my prompts and task, it passed despite the 2 bits, the same as 6 of the other models.

Bolt also comes with an official small draft model (drafter) for speculative decoding. Adding it with --spec-type draft-dflash, as in the official instructions, short answers came out at 105–115 tokens/s, about 2.5× the 42.7 without drafting. Longer answers in Japanese stayed at 55. The error-fixing task took 6.0 s without drafting and 3.6 s with it.

With 4 requests at once, the total went up

I started llama-server with -np 4 and sent the same prompt 4 times at once. I measured only 3 models.

Each request gets slower, but the total was about 1.5–2× a single request. Qwen3.6's per-request numbers vary because its answer lengths varied, from 267 to 421 tokens.

Fast and able to finish the task were different things

The fastest model, Flint, did not fix the error in 12 steps. It never opened config.yaml. It ran commands like pwd, echo hello and ls -la /, and ended up just looking around at folder contents. Flint wrote out its thinking even when I passed the setting to turn thinking off. Because of that, on the code prompt with an 800-token answer limit, it hit the limit on thinking alone and the answer was empty. With the limit at 4,000 it passed (3,408 tokens), but it still didn't fix the task. In tool calling, it called ls -la before the correct command (both times, at the 800 and the 4,000 limit).

Hermes 4.3 didn't fix it either. In what the log kept, it repeated npm start, npm test and npm run build (only the end of the log survived, and the final verdict line is cut off too).

The other 6 fixed it, in different numbers of steps:

• 5 steps: Qwen3.6-35B-A3B, Bolt, Super VL. List files, read the config, fix it, run the check command: close to the shortest path.
• 8 steps: Mistral Small 3.2.
• 9 steps: Qwen3.8-27B. Its report included the cause (a capital O where a 0 should be).
• gpt-oss-120B: fixed it correctly at step 5, but then didn't run the check script (check.py) and kept listing folders until it used up all 12 steps. Its report was empty.

On the JSON prompt, 4 models (Bolt, Super VL, Flint, Hermes 4.3) returned bare JSON, and the other 4 wrapped it in a Markdown code fence tagged json. With the fence removed, all 8 parsed. If you pass the output straight to another program, you'll want a step that strips the fence.

With Hermes 4.3, GPU memory grew by about 7GB at the 32k context (23.5GB → 30.7GB). The other models grew by 0.2–4.5GB. If you use it with long contexts, budget more than the model file size.

Commands you can reuse

Speed (llama-bench):

llama-bench -m <model>.gguf -ngl 999 -fa 1 -p 512,4096 -n 128,512 -r 3 -o csv
# Long-context row
llama-bench -m <model>.gguf -ngl 999 -fa 1 -p 32768 -n 128 -r 1 -o csv
# Move expert weights to the CPU side (MoE only; the number is the layer count)
llama-bench -m <model>.gguf -ngl 999 -fa 1 -p 512 -n 128 -r 1 -ncmoe 8

Server (llama-server):

llama-server -m <model>.gguf -c 32768 -np 1 -fa on -ngl 999 --jinja --host 127.0.0.1 --port 8080
# For models that handle images, add that model's mmproj
#   --mmproj <mmproj>.gguf
# 4 requests at once
#   -np 4

To turn thinking off, I put "chat_template_kwargs": {"enable_thinking": false} in the request body. gpt-oss can't turn thinking off, so I used the lightest setting, {"reasoning_effort": "low"}, and set the answer limit to 1,500 tokens.

Bolt with speculative decoding (from the official model card's instructions):

llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf -c 32768 -fa on -ngl 999 --jinja \
  --spec-type draft-dflash --spec-draft-n-max 7 \
  --spec-draft-model NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --spec-draft-ngl 999

What I didn't measure

• One PC only. I didn't compare with other machines.
• Quantization isn't aligned. This is not a same-bit-width comparison.
• I judged capability with only 5 short prompts and 1 task. I did not judge the quality of the Japanese and image answers. All prompts and the task were run once each at temperature 0.
• For the ~87GB Super VL: where the part that didn't show up in nvidia-smi was placed. Also the server prompts and 32k context on that version.
• I didn't measure with tools other than llama.cpp, such as Ollama. (The difference from Ollama on the same file is in the previous article. There, Ollama used speculative decoding by default, and Qwen3.6 ran at about 100 tokens/s.)

Takeaways

• On this PC (about 82GB of GPU memory), generation speed lined up by the number of parameters active per token, not by total parameter count. The 117B gpt-oss-120B was about 4× faster than the 27B Qwen3.8-27B.
• A model of about 87GB still ran with everything assigned to the GPU. nvidia-smi's ~82GB was not the upper limit of what could be placed.
• Speed and finishing the task were separate. The fastest model, Flint, could not fix a one-character error in a config file.

Licenses: of the 8 models, 6 are Apache-2.0, and the two Nemotron 3.5 models are OpenMDW-1.1. OpenMDW-1.1 asks you to keep a copy of the license and the attribution notices when you redistribute the model, and does not restrict how the outputs are used.

About this series

This is part of my series on the Surface Laptop Ultra (Windows on Arm with an NVIDIA GPU). Related articles:

• Initial setup: on Arm Windows, check whether each tool can use the GPU before installing it (Japanese)
• Ollama vs. llama.cpp on the Same GGUF: The 15% Speed Gap Was the Defaults, Not the Engine (English)
• Moving ComfyUI from Intel Arc to Arm WSL2: The Same requirements.txt Didn't Install the Same Versions (English)

The original of this article: 120B が 27B より約 4 倍速かった。Surface Laptop Ultra でローカル LLM 8 種を同じ条件で測る.