When the Model Has to Change: Four Problems That Tell You It’s Time
From document extraction and computer vision to LLM fine-tuning: four engineering problems that taught me when to train, when to adapt, and when to leave the model alone.
When a model gives poor results, retraining or fine-tuning is rarely the right first move. The cause is usually upstream: the data, the retrieval, the preprocessing, the prompt, or the code around the model.
But there are problems no amount of surrounding engineering will fix. The model itself has to change. In document AI I’ve met four of them, and they look different enough that each needs a different response.
Four problems that mean the model has to change, and the response to each.
Problem 1 (2021–22): your documents need a model that learns from your own data
The signal: off-the-shelf extractors work on common documents, but every domain has its own layouts and extraction requirements. Rule-based configurations become hard to maintain, and word-classification approaches need expensive word-level annotation.
What works: an end-to-end extraction architecture that learns directly from documents and their expected extracted values.
One example is Attend, Copy, Parse (ACP), from a 2018 paper by Rasmus Berg Palm, Florian Laws and Ole Winther. What makes it interesting is the training approach: instead of annotations marking every relevant word, it learns from the document image, the OCR words, their positions on the page, and the final expected value.
It combines four components in one end-to-end network, trained together rather than as separate steps:
- Attend: learns where to focus, combining image features, OCR text and spatial information.
- Copy: uses the attention to retrieve the relevant word sequence (N-gram) from a memory representing the document.
- Context: computes a representation of the attended part of the document, to help interpret what was copied.
- Parse: combines the copied text and the context to produce the required, normalised output.
Take the date 02/03/2018. Depending on the document's locale, it means 2 March or February 3. The parser uses both the copied text and its contextual representation to interpret the value.
From my work: this was my first experience training document-extraction models. On a form-extraction product covering health-insurance, loan and appraisal forms, I started with UI and backend development, then moved into preparing the training datasets, training the ACP models, integrating them into the backend and evaluating extraction quality field by field.
Problem 2 (2022–23): the model architecture becomes harder to scale than the problem itself
The signal: each new field, document type or customer request means another model to train, serve and monitor. Accuracy is fine; the approach is what doesn’t scale.
What works: stop training by default. Start from pretrained open-weight models where they already do the job, use plain engineering where a model isn’t needed at all, and train only where nothing else fits.
From my work: the ACP design, like the paper, used one model per field: patient name, loan amount, date. Every new field meant new data, a new training run and a new model to serve. So the next release stopped training ACP models. Key-value extraction moved to a pretrained open-weight model with a transformer backbone, which handled new fields without a new model each time, and templates with ready field sets for common document types sat on top.
The lesson: don’t train a separate model for every new field. Reconsider the architecture, and reuse pretrained capabilities wherever they’re good enough.
Problem 3 (2023): visual elements need different solutions
The signal: the information isn’t in the text. It’s in a table’s structure, a ticked box, a photo or a face. A text model, or OCR, can’t see it, and pretrained detectors or public datasets don’t cover it reliably on your real documents.
What works: decide per visual element. Some need a trained model, some need only engineering, and the deciding question is whether classical computer vision is reliable enough on its own.
Get Dineshk’s stories in your inbox
Join Medium for free to get updates from this writer.
From my work: on a production document platform, four kinds of visual element each got a different answer:
- Tables: a pretrained table model wasn’t enough on the documents, so I explored training a DETR (detection transformer) model for table detection and structure.
- Images and faces: handled with classical computer vision rather than a trained model. CV-based detection was reliable enough to find, crop and clean photos and faces, so there was nothing to train.
- Checkboxes: the hardest one, and the one that needed real training. Whether a box is ticked can decide an answer on a medical or insurance form. OpenCV contours, OCR-based approaches and pretrained YOLOv5 and YOLOv8 detectors weren’t reliable, and the public datasets I could find had no suitable checkbox annotations for the task.
For checkboxes, the dataset had to come first. I built one from full-size scans and split the problem in two. A Detectron2 Faster R-CNN handles detection: where each checkbox is. A separate classifier handles state: whether it’s ticked. They’re different tasks, so they were handled by different models. Much of the effort went into the data itself:
- converting annotations between CSV, COCO and YOLO formats
- fixing coordinates that weren’t normalised or fell outside the image
- fixing category-ID mappings, duplicate annotation IDs and image-size mismatches
Those bugs are the dangerous part: training finishes without errors while the model learns from wrong labels.
Problem 4 (2024): retrieval or prompting keeps failing, and fine-tuning looks tempting
The signal: an LLM keeps producing the wrong output despite better retrieval and tighter prompts, and fine-tuning starts to look like the answer.
What works: first ask what kind of failure it is.
- If the model behaves inconsistently on a narrow, repeated task (format, style, which field to pick), fine-tuning can help.
- If the model lacks a fact it can’t see (an ID that must be looked up, information that changes), fine-tuning won’t help. That’s a job for retrieval, tools or code.
From my work: in the voice agent from Part 3, RAG over API documentation kept picking the wrong endpoints.
- Hypothesis: the model wasn’t adapted enough to the API to understand it and choose the right endpoint.
- Experiment: fine-tune a 4-bit Mistral-7B-Instruct with Unsloth and LoRA / QLoRA, on about 1,200 question-answer examples from Stack Overflow kept in PostgreSQL, served through FastAPI, with checkpoint-based recovery for runs interrupted on a shared GPU server.
- Finding: it gave a repeatable pipeline from dataset to served model, but it couldn’t supply what the calls actually needed: exact, current endpoint details and IDs that have to be looked up.
- Decision: let the LLM interpret the request, and let deterministic code select and call the API.
The hypothesis was reasonable and wrong, and testing it is what made the real fix clear. Before fine-tuning, check the platform too: model support, input types and token cost can rule an approach out before any data is built.
How I decide whether a model needs to change
After these experiences, I approach model-related problems with five questions:
- Does the model fail to adapt to your document domain? Consider an architecture that learns from your own document data.
- Are you maintaining too many models? Reconsider the architecture and reuse pretrained capabilities.
- Is the information visual rather than textual? Choose between classical computer vision and training a specialised model.
- Is the model inconsistent on a narrow, repeated task? Evaluate fine-tuning.
- Does the model lack current information or need exact lookups? Use retrieval, tools or deterministic code instead.
Across all of these, one thing stayed the same: the hardest part was rarely the training command. It was preparing the data, validating it, and finding the errors that didn’t make anything crash.
The most useful question isn’t “which model should I use?” It’s “what exactly is preventing the existing model from solving this problem?” Sometimes the answer is the architecture. Sometimes it’s the data. And sometimes the model doesn’t need to change at all.
Next in the series: bid agents. Multiple agents on one long task, document extraction at the core, and the evaluations that kept the results honest.
