Abliterated Gemma 4 and GPT-OSS 120B on Ollama: how GPU offload layers decide what fits - DEV Community

DEV Community

AI-KSK
AI-KSK

Posted on

Abliterated Gemma 4 and GPT-OSS 120B on Ollama: how GPU offload layers decide what fits

#comfyui #ai #tutorial

This episode is not a ComfyUI graph walkthrough — it is a local-inference episode about running refusal-removed ("abliterated") large language models through Ollama, and about the one configuration knob that decides whether a 120B model runs at all on a single consumer GPU: the number of model layers Ollama pushes onto the GPU.

Evidence note: this episode ships no workflow JSON and no workflow links (工作流链接：无 in 说明.txt), so there are no graph parameters to quote. Everything below comes from the episode subtitles, and the performance numbers are what the author reports on his own machine — they are not an independent benchmark.

Where this fits

You have one big GPU (the author uses a rented 5090) and you want a local chat model without refusal behaviour, without a cloud API, and without rebuilding an inference stack. Ollama plus Open WebUI gives you the chat surface; the offload-layer count gives you the control over how much of the model lives in VRAM versus system RAM.

What the episode covers

  • Two abliterated models: Gemma 4 (the small, fast option) and a GPT-OSS large model at 120B (the "still top-tier even after unlocking" option).
  • 5090 startup presets for both, so the deployment is one click.
  • A written configuration document that explains how to set the GPU offload layers in an Ollama model file, so that a large model can be squeezed onto less VRAM.
  • A working Open WebUI session on top of the deployed instance.

Step by step

  1. Deploy an instance. The author's route is the Compshare (优云智算) image: open the image, click deploy, wait. The image is self-starting, so there is no launch command to paste — that is the difference from the JupyterLab-plus-terminal images used in his ComfyUI episodes. Registering through the referral link is described as bringing some free runtime.
  2. Pick the GPU. The 5090 is called the most cost-effective choice for Gemma 4, and the 5090 preset is described as more stable and more "one-click".
  3. Open the chat surface. Click Ollama in the instance panel to reach Open WebUI, then log in with the instance's unified e-mail/password.
  4. Choose a model at the top of the UI. The model dropdown includes both the plain models and the "preset" variants.
  5. Understand what a preset is. The preset entries are not separate models — they are the same weights with a preset file attached and a new display name. This matters when you are comparing timings: you are comparing presets, not different checkpoints.
  6. Tune offload layers when the model does not fit. Open JupyterLab, edit the Ollama model file so the GPU layer count matches your VRAM, and re-create the model. The author's point is that this is the lever that lets a lower-VRAM card run a much larger model; he publishes his notes and says that handing the document to an LLM gets the layer count right in the large majority of cases.

Reported numbers

  • GPT-OSS 120B: about 11 seconds for a response, with 67.7 GB of memory in use. The author calls this "not fast at all, but acceptable".
  • Gemma 4: 5 GB of memory in use, because most of the model stays in VRAM; the author describes the thinking speed as very fast.
  • Both are his own readings from the running instance, at one point in time, on one machine.

Limits

  • Abliterated models remove refusals; they do not make the model smarter. The author says so directly: "maybe not that clever, but absolutely free".
  • Offload layers buy capacity by spending speed. Anything that does not fit in VRAM runs from system RAM.
  • The episode gives a qualitative recipe for the layer count, not a table of numbers per model size.
  • The configuration document lives outside the video (the author points to a cloud-drive link in his Bilibili description). No Bilibili link and no mirror link are recorded in this episode's 说明.txt, so neither the document nor the image is reachable from this package.
  • Timings are single observations and are not reproducible without the same image, preset and hardware.

Episode resources

Top comments (0)