Fine-Tuning Small Models for On-Device Agents: Train, Convert, and Measure What Changed
How litetune rebuilds the LoRA-to-device workflow and measures the model that actually ships.
In February 2025, I published a ten-step guide to fine-tuning Gemma with LoRA in Colab, converting the model, and running it on a phone. I build Flutter Edge AI (formerly flutter_gemma), a plugin for running these models directly on the device, so fine-tuning was the natural next step.
Today, following that guide step by step no longer works. I first tried to update it, but replacing one stage led to another. LoRA was the only part that stayed the same.
So this is not a patched version of the old tutorial. It is a new workflow, with one stage the original did not have: verifying the converted model on held-out data.
What Changed in the Workflow
The old guide was built around the first Gemma models — 2B and 7B — and MediaPipe for on-device inference. The entire path from training to a MediaPipe-ready artifact fit inside a single Colab notebook. Today, its parts are spread across Hugging Face, TRL, litert-converter, and LiteRT-LM. They have different dependencies, and export settings vary by model architecture. LoRA training itself still looks familiar — the complexity now sits around it.
Before training, you need to validate the data and create and record a held-out split. After training, you need to merge the adapter, select the export settings, build a .litertlm file, run it through LiteRT-LM, and compare its outputs with the original model. A working checkpoint does not guarantee that the final artifact will behave the same way.
When it became clear that the old notebook could not simply be updated, I decided to put the modern workflow into one tool. That is how litetune started: an open-source command-line tool, released under Apache 2.0, that takes a dataset all the way to a verified .litertlm.
Its job is to remove the handwritten glue code between a dataset and a model that an application can run. Instead of maintaining separate scripts for data preparation, LoRA training, conversion, verification, and packaging, you provide the model, dataset, and task and invoke the corresponding litetune stages:
The most important addition is verification. The old workflow ended when the converter produced a file. litetune runs the converted .litertlm through LiteRT-LM, compares it with the float model on held-out data, and only then packages the bundle.
You no longer need to write a separate evaluation harness, preserve the outputs of two models by hand, or implement the paired comparison yourself. verify performs that work and stores the result together with the conditions under which it was measured.
- prepare validates each dataset row and creates a deterministic training and held-out split before training begins.
- tune fine-tunes the model with LoRA on the training split and saves the resulting adapter.
- convert exports the tuned checkpoint as a .litertlm file using the settings required by that model architecture.
- verify runs the converted model through LiteRT-LM and the float reference through PyTorch on the same held-out examples, then reports their paired difference and its uncertainty.
- bundle packages the converted model with its task contract, declarations, measurement report, and the metadata needed to reproduce the result.
litetune creates an isolated environment for each stage, so training, conversion, and inference can use the dependency versions they require without users reconciling them by hand.
Six fine-tuned models have completed this workflow: FunctionGemma, Gemma 3 270M, Qwen2.5-0.5B-Instruct, Qwen3-0.6B, Gemma 3 1B, and Gemma 4 E2B. Their architectures and results differed, but the path from dataset to measured .litertlm remained the same.
Not every model is trained to produce the same kind of answer. In the Banking77 runs, the target is a single intent label, so verify uses the exact-text scorer. FunctionGemma has a different job: it must choose a tool and produce structured arguments, so verify uses the tool-call scorer. The task contract tells litetune what a correct answer looks like, while the rest of the workflow remains the same.
Banking77 is PolyAI’s open dataset of 13,083 questions customers ask an online bank, each labeled with one of 77 intents, such as
card_arrivaloractivate_my_card. The dataset on Hugging Face.
On macOS, install litetune through Homebrew:
brew install DenisovAV/tap/litetuneOn Linux, including Jupyter notebook environments such as Google Colab, install it through pip:
pip install litetunelitetune runs the full pipeline on Linux and macOS. Training and float-reference evaluation use CUDA or Apple MPS when available, with CPU as a fallback. The commands remain the same across platforms. On Windows, prepare, tune, verify, and bundle are already available. convert is blocked by the missing upstream Windows converter package. Windows will join the fully supported platforms when litert-torch#968 is resolved. If Windows support matters to you, add a thumbs-up to that issue.
Why This Matters More for an Agent
An agent is not a single request to a model. It is a loop:
Every step requires another model invocation. Running locally removes a network round trip from each iteration and keeps the loop available offline. But a small model must perform its narrow task reliably: one mistake in a function name or argument can stop the entire process.
FunctionGemma is a good example. It is a 270-million-parameter model designed to turn a user request into a function call. Google’s model card reports an improvement from 58% to 85% after fine-tuning on its own benchmark. In our test of 640 held-out requests, the base model scored 0.7625 and the fine-tuned model scored 0.9141. Both used the same dictionary-sorted representation of the function declarations.
That detail matters. Changing the declaration order alone can change the model’s outputs, so we cannot compare the fine-tuned model’s best result with the base model’s worst result and attribute the entire difference to training. But the float model’s score still does not tell us how the converted artifact performs. Export, quantization, packaging, and a different runtime still stand between the checkpoint and the file that reaches the application.
The Converted Model Is What Needs Verification
After training, we know how the float checkpoint performs. But the application will not run that checkpoint — it will run the converted .litertlm file. That is the artifact that needs to be verified through LiteRT-LM.
Get Sasha Denisov’s stories in your inbox
Join Medium for free to get updates from this writer.
The simplest check is to inspect one of its outputs. This can reveal an obvious failure, but it cannot tell us how often the model is wrong. The converter’s exit code does not help either: it confirms that the file was created, not that the converted model still solves the task.
This is why verify takes the held-out dataset, the scorer, the converted model, and the float checkpoint from which it was created. It runs both versions on the same examples and compares their outputs pair by pair.
Here is the result of verifying FunctionGemma on LiteRT-LM 0.17.1:
A positive difference means a loss relative to the float model. The interval does not cross zero, so litetune marks the result as resolved: with this sample, the measured difference can be distinguished from random variation.
verify deliberately compares two real execution paths. The float checkpoint runs through PyTorch, while the converted .litertlm runs through LiteRT-LM’s native tool path — the same path Flutter Edge AI uses for FunctionGemma. This is not a laboratory estimate of quantization alone. It answers a practical question: does the converted model still solve the same task? The devices, backend, runtime version, and other conditions are stored with the result.
Quantization: What It Is and Why It Matters
A model that runs on a phone has to fit both the device’s storage and its available memory. In the original checkpoint, every one of the model’s hundreds of millions — or billions — of weights is typically stored as a 16- or 32-bit floating-point number. Together, those weights can take hundreds of megabytes or several gigabytes.
Quantization stores those weights at a lower precision, commonly using eight or four bits per weight. This makes the converted model smaller and reduces the amount of memory needed to run it, which makes on-device inference morepractical. But the values have to be approximated, and that approximation can change the model’s outputs.
Different quantization recipes decide which parts of the model to quantize and how to group their weights. Two artifacts with the same bit width can therefore preserve quality differently. The only way to know what happened is to test the converted artifact — which is what verify does.
After 8bit Quantization, the Models Stayed Close to Float
The FunctionGemma result above shows that verify can detect a measurable difference. But one model does not tell us how often quantization changes the result. I therefore verified five other fine-tuned models, converting each with two eight-bit recipes. Every artifact was tested on the same 600 held-out examples as its corresponding float model.
All ten comparisons were unresolved on LiteRT-LM 0.17.1. Here, that is a good outcome. A resolved result would mean that verify detected a measurable gain or loss after conversion. Unresolved means that the observed differences were too small to distinguish from zero: the eight-bit models stayed close to their float references, and verification found no detectable degradation.
This does not prove that the two versions behave identically. It means that, on these held-out samples, conversion did not produce a measurable loss.
4bit Changed the Picture
The same conclusion did not hold at four bits. For Qwen2.5–0.5B, two standard channel-wise recipes reduced exact match by 25 to 27 percentage points. Two block-wise recipes preserved considerably more quality, but still lost about eight points. verify marked all four differences as resolved: these were measurable regressions, not statistical noise.
Nor was there one four-bit recipe that worked safely across architectures. On other models, some candidates were refused by quality checks before or during full verification. Among the block-wise conversions that completed, the measured loss ranged from 2.83 to 40.67 percentage points, depending on the model and recipe.
The same bit width did not produce the same result across models or recipes. Eight-bit conversions stayed close to float, while four-bit conversions ranged from moderate degradation to unusable output.
Refused does not mean that conversion failed: the file was created, but litetune would not accept it as a verified artifact. Repetition means that the model’s output collapsed into repeated text and crossed the liveness threshold, either during the preflight gate or the full held-out run. Timeout means that the candidate could not finish the five-prompt preflight gate within its 900-second budget. In both cases, litetune reports no quality score rather than treating an incomplete or unusable run as a valid result.
Four-bit quantization therefore requires care. A successful conversion can still produce a model that is much less accurate, repeats itself, or is too slow to finish. Treat every recipe as unverified until you have run the converted artifact on held-out data. Otherwise, a smaller file can reach the application with outputs that are effectively unusable. Convert the model with several recipes, run verify, and choose the artifact by its measured behavior rather than the name of the recipe.
From a Verified Artifact to the Device
The result of this workflow is a .litertlm file verified through LiteRT-LM on held-out data. litetune prepares and measures the model; the next step is to put that same file into an application.
For a Flutter application, this completes the path that motivated the article. Flutter Edge AI supports several local inference engines, and LiteRT-LM is one of them. When the application is given a .litertlm file, Flutter Edge AI runs it through LiteRT-LM—the same runtime path litetune used to verify the converted model.
The file produced by litetune can be bundled with the application, downloaded from your own server or Hugging Face, or opened from local storage. One Dart API is used to work with the model on Android, iOS, macOS, Windows, Linux, and the Web, while the platform-specific LiteRT-LM implementations remain under the hood. No additional conversion or Flutter-specific model format is required. The .litertlm itself is not tied to Flutter: it is a native LiteRT-LM bundle.
If the application is written natively, the same file can be integrated directly. LiteRT-LM provides a Kotlin API for Android, a Swift API for iOS and macOS, and a cross-platform C++ API for desktop and embedded applications. Each option runs the same .litertlm through LiteRT-LM.
To try the model quickly on Android without building an application, import it into Google AI Edge Gallery. Copy the .litertlm to the device, add it from the model screen, and run it like any other model in the gallery.
For a text model, that is enough for a first on-device check. If the model was trained to call functions, the application must provide the available tool declarations, execute the returned call, and continue the agent loop. Flutter Edge AI exposes one API for that workflow; in a native application, the same logic is connected through the corresponding LiteRT-LM SDK.
What These Results Establish — and What They Do Not
The results in this article are not a general model leaderboard or a universal rule for quantization. They cover six fine-tuned models, but only two evaluated workloads: intent classification on Banking77 and function calling on mobile-actions. The eight-bit finding applies to the models and recipes measured here; it does not mean that every eight-bit conversion will automatically preserve quality. The same is true at four bits: the measured losses cannot be transferred to a model or recipe that has not been through verify.
litetune can train any task expressed as prompt → completion, but verify currently provides two evaluation contracts. exact-text works for classification and other tasks with one canonical textual answer. tool-call checks the function name and its argument values. That is not enough for free-form tasks with several valid answers, including translation, summarization, information extraction, question answering, code generation, or planning a sequence of actions.
The next step is to make those tasks first-class parts of the entire workflow, not merely allow their completions to be used for training. Each task needs its own data contract, input validation, rules for producing the training completion, and a scorer that measures what correctness means for that task. Structured JSON could be checked against a schema and field values; extraction could use field- or token-level F1; code could run in isolated tests; planning and multi-tool agents could be scored on the sequence of calls, their arguments, and the final answer. Translation, summarization, and other free-form outputs will need a reproducible semantic scorer whose model, version, and prompt are recorded with the result.
Every verify result also belongs to the conditions of its run: the held-out split, scorer, runtime version, backend, devices, and generation settings. litetune stores those conditions with the result because a number separated from its experiment cannot be reproduced or compared correctly.
Quality verification does not replace performance measurement on the target device. A model may preserve its answers but exceed the available memory, run too slowly, or behave differently across CPU, GPU, and NPU backends. After verify, the selected bundle still needs to run on the phone or computer where it will be used, with latency, memory use, and energy consumption measured there.
Where to Start
If there’s one thing to take away, it’s that the converted model is not the checkpoint you trained. Until it has run on held-out data, you don’t know what you’re shipping. My advice, in order. Start with an eight-bit recipe and run verify on your own held-out data. Eight bits cost little on every model measured here, but yours is not on that list. Move to four bits only when size forces you to: convert with several recipes and keep the artifact verify accepts, not the one with the most promising name. Measure latency, memory and battery last, on the device the model will actually run on. Don’t ship the smallest file because a recipe produced it. Ship the smallest one that passed.
This is only the beginning. I’ll keep adding model architectures, conversion recipes and platforms, and I’m still deciding which kind of task comes next. A few candidates are on my mind: several tool calls in a single turn; the full agent loop, where the model sees a tool’s result and decides what to do next; and typed decisions, where the model returns a probability for each option instead of text — the idea behind TypeSafe AI’s Jev. But I’d rather build what you actually need, so tell me in the comments which of these, or something else entirely, would help you most.
If you are working with a model or task that litetune does not support yet, try the tool and tell me what is missing. Real datasets, reproducible examples, new scorers, model recipes, and results from physical devices are especially useful. Issues and pull requests are open in the litetune repository, and contributions are welcome.
