The Cheapest LLM Call Is The One You Skip
Right-sizing an AI document pipeline without giving up accuracy
By: Pruthvi Dondapati, Venkatesh Alapati, Vamsikrishna Vallivedu
AI assistants made it easy to put a large language model in the middle of a document pipeline. They did not make it easy to run that pipeline at volume. The first version of anything LLM-based tends to reach for the most capable model available and use it everywhere, on the assumption that it is less likely to produce an inaccurate extraction than a smaller, cheaper model would be. It is also a bill for the model charges that grows in a straight line with your traffic.
We built and tested a pipeline that extracts structured data from healthcare claims documents. The pipeline reads those documents, extracts the fields that matter, and cross-checks them against authoritative source systems. Discrepancies are surfaced to human reviewers for follow-up; the pipeline flags candidates for review.
We ran it at high, sustained volume, and eventually had to answer the question every LLM-heavy pipeline must answer. If volume keeps growing, does cost have to grow with it?
The answer was no. What follows is the set of levers that got us to a cost model that does not grow in lockstep with volume, what each one actually did, and the measurement discipline that let us trust the result instead of hoping for it.
Not every step needs the same model, or a model at all
The instinct when you build an LLM pipeline is to reach for the most capable model and use it everywhere. That instinct is correct for getting something working. However, it is the wrong default once that pipeline is running at sustained volume.
The fix that worked was not “use a cheaper model everywhere.” That trades cost for accuracy risk, in exactly the cases where you can least afford it, because the complex documents are the ones the cheap model fails on. The fix was tiered routing. Attempt the cheapest sufficient method first, and route only as far as a given document requires.
The important part is the tier that has no model in it at all.
1. Deterministic parse, zero tokens
The first routing tier never calls a model. It uses the document’s own text and layout geometry. Claim pages are located by reference-number pattern. The detail table is parsed by column-anchor geometry rather than by guessing at whitespace. Balance and payment amounts, provider numbers, and remark codes come out through pattern matching.
Then the tier proves its own work with cross-field arithmetic checks against the values it just extracted. Do the line items reconcile to the stated totals? Our set of checks grew to eight. Two representative ones:
Representative arithmetic checks
billed == allowed + not_covered
balance == allowed - paymentThis is the part worth stealing. A deterministic pass does not need to be clever, and it does not need to handle every document. It needs to know when it is wrong. Arithmetic self-checks are how a non-AI tier earns the right to skip the model. If the numbers reconcile, you have a correct extraction and you did not make a network call, spend a token, or wait on a queue.
A large share of documents pass every check here and never touch an LLM.
2. Smaller model, text only
When the deterministic pass fails for a reason other than the arithmetic check, the document falls back to a smaller, cheaper model from the same family as the one used for the hardest cases.
One detail matters more than the model choice. This tier does not send an image. It builds a monospace text “layout snapshot” of the page, which preserves the spatial arrangement the model needs to read a table, and sends that as text. Vision tokens are about 3x more expensive than text tokens for the same page, at current pricing. Rendering the page as characters instead of pixels keeps a spatially complex document on the cheap side of that line.
3. Larger model, full page image
Only when the cheaper path genuinely cannot do the job does a document reach the largest model with the full-page image attached. Rotated pages, multiple claim identifiers on a single page, and outright failure of the arithmetic check route here directly, because we already know the tiers below will fail on them.
The point is not that a small model is always good enough. The point is that the tier is a decision made per document and per failure mode, not a blanket policy. The expensive tier is reserved for the cases that earn it by failing the cheaper ones.
One practical addition: log which tier resolved each document and export it. Tier hit rate is the only way to know whether your ladder is working or whether everything is quietly falling through to tier three.
Don’t pay for a second call to check the first one
Our most expensive mistake wasn’t choosing the wrong model — it was calling it too many times.
An earlier version of the extraction step made two separate LLM calls per document. The first sent the document’s text and asked the model to extract fields. The second was a vision call that took a rendered image of the same page along with the first call’s output and asked the model to validate it.
The double spend was not accidental. It was tracked. The code summed both calls on every run:
Tracking both calls
total_input_tokens = input_tokens + validator_input_tokens
total_output_tokens = output_tokens + validator_output_tokensEvery document paid for a text call and a vision call. The vision half was the expensive half.
A checking call is a natural thing to build. If the model returns free-form text, you genuinely do need something to confirm it came back in the right shape with sensible values. The trap is treating that as fixed and then trying to optimize the checking call instead of questioning it.
We asked why the second call existed rather than how to make it cheaper. It existed to catch a class of error: the model returning malformed or implausible output. Structured output removes that class of error at the source.
BEFORE: a text extraction call, then a vision validation call on a rendered image of the same page.
AFTER: a local, non-AI text extraction step feeds a single schema-constrained call. The model returns data conforming to an explicit, typed schema. Not prose that has to be parsed and checked afterward.
Single schema-constrained call
# One call. The schema is the contract, so there is nothing
# left for a second call to verify.
response = client.chat.completions.parse(
model=EXTRACTION_MODEL,
messages=build_messages(locally_extracted_text),
response_format=EOBExtractionModel, # typed schema
)
fields = response.choices[0].message.parsedTwo things happened here.
First, the local text extraction step replaced an AI call with a lightweight library call. No tokens, no network, no latency. Second, the schema-constrained response replaced the validation call with a type contract. The model cannot return a shape the schema does not allow, so there is nothing left for a second opinion to catch.
Across production traffic, this alone cut the number of LLM calls per document by roughly half. That is not a rounding error. It came from asking “why does this call exist?” instead of “how do we make this call cheaper?”
Prompts and models are separate levers, so use both
It is tempting to treat “which model” as the only cost dial. It is not.
In one part of our system, a classification task with a small, structured output, we made two changes days apart. First, we moved the task from the larger model to the smaller one. Separately, we rewrote the prompt to be shorter, removing repeated instructions and formatting overhead without changing what the task asked for. That second change let us cut the maximum output allowed for the call by roughly two-thirds, from 1500 tokens to 500.
Both changes reduce cost. Neither replaces the other. A tighter prompt reduces cost regardless of which model answers it. A smaller model reduces cost regardless of how verbose the prompt is. Treating them as one lever, “switch models,” leaves savings on the table even after you have made the harder decision to right-size the model.
Get Evernorth Tech Blog’s stories in your inbox
Join Medium for free to get updates from this writer.
The max-tokens ceiling is the quietly useful one. If a task genuinely produces a short, structured answer, an oversized ceiling costs you nothing on most calls and a lot on the pathological ones where the model rambles.
Make model choice a governed decision, not a habit
None of the above sticks if the model name is a string copy-pasted across call sites. Ours went through six recognizable stages, and the arc is more useful than any single step.
We started with a hardcoded model name at three separate call sites and no token instrumentation at all. Model choice was not a decision. It was whatever the code already did, and nobody could see what any feature cost.
Instrumenting token usage came first, and it is the prerequisite for everything after it. Then the hardcoded string moved into configuration, which made the model swappable without a code deploy.
Configurable is not the same as reliable. That stage brought its own failure mode: a missing or wrong environment variable can silently regress every call to some default, and we had a hotfix where a hardcoded value had to go back in temporarily to unblock a deploy. A config value with no source of truth behind it is a guess with better ergonomics.
The step that fixed it was a model factory that reads deployment names from a single, centrally managed configuration source instead of a hardcoded string, behind a small enum:
Governed model selection
class ModelType(Enum):
LARGE = "large" # frontier-capable
MINI = "mini" # small, fast, cheap
LATEST = "latest"
model = ModelFactory.get(ModelType.MINI) # one governed decision pointWith a factory in place, per-task model choice became a one-line change instead of an archaeology project. Different tasks got pointed at different models based on what they actually needed, with the reasoning written into the code comment next to the choice. Finally, direct SDK calls moved behind a shared gateway with standardized model group names rather than per-team deployment names.
That last step is worth framing honestly. It is governance and standardization, not a direct cost reduction. It does not change which model a given call uses. It changes how that choice is expressed and enforced, which makes future model swaps a config change instead of a code change across every team building on the same gateway.
How to validate a change like this without gambling on production
None of this is worth much if it quietly makes the pipeline less accurate. If there’s one thing we encourage other teams to copy, it’s the methodology we used to avoid that. It is not specific to LLMs. It is disciplined benchmarking applied to a place people often skip it.
Benchmark multiple candidate approaches on a modest sample
We compared four approaches on an initial sample of representative records, scoring each against the existing production pipeline’s own output and against the underlying source-of-truth data, not just against each other.
- Low-resolution image plus LLM. Cheapest thing we tried. Around 33% accuracy. Rejected outright.
- A markdown-conversion variant. Around 97.9%. Discontinued for being slightly behind the winner.
- High-resolution image plus LLM. Around 98% accuracy, high cost and latency.
- Local text extraction plus LLM. Around 98.7% accuracy. The winner.
The low-resolution option is the instructive one. It used the fewest tokens of anything we tested and it was never a real candidate. Token minimization was never the goal. Accuracy at acceptable cost was.
Reject anything that does not meet or beat the current baseline
Cost and latency only become interesting once accuracy is off the table as a trade-off. If a cheaper approach is also less accurate, it is not a cost optimization. It is a quality decision wearing a cost optimization’s clothes, and it should be argued on those terms.
Re-validate the winner on a meaningfully larger sample
We re-ran the winning approach on a substantially larger set of representative records before shipping, to be sure the small-sample result was not a fluke. Always manually review a sample of where the new pipeline disagrees with the old one, not just the aggregate accuracy score. A subtle regression hides in the disagreements, not in the average.
Re-confirm gains after you ship
After the change shipped, we compared billing and latency metrics from comparable processing windows before and after deployment. This was a
post-deployment evaluation against operational metrics from production workload samples, not a held-out lab test set. A lab benchmark tells you a change should work. Only a post-ship evaluation against actual data tells you it did.
The results, without the spreadsheet
On the benchmark, moving to the tiered, structured-output approach cut average token consumption by roughly 22%, average latency by roughly 62%, from about 14.5 seconds to about 5.5 seconds per record, and average cost per case by roughly 87%. Accuracy held at 99.4% measured against the prior vision-based pipeline’s own output and improved modestly against the underlying source-of-truth data.
In a post-deployment evaluation of operational billing and latency metrics across comparable windows before and after the change, measured on production workload samples rather than a held-out test set, we saw a reduction in per-case cost of roughly 86% and a reduction in average latency of roughly 55%. In line with what the benchmark predicted.
Through tiering, call elimination, prompt compression, and deliberate model governance, actual AI spend for this work came in under 5% of the budget set aside for it, without giving up the accuracy bar the original, more expensive pipeline had set.
There is one lever we have identified and not yet pulled. Batch API pricing on the same workload would plausibly cut costs in half yet again on top of what is described here. The benchmark infrastructure and baseline numbers already exist. The next step is running the same workload for comparison.
What to take from this, regardless of your stack
Model choice is a per-task decision, not a project-wide default. The task that needs a frontier-capable model and the task that needs a lightweight classifier are rarely the same task.
Before adding a second LLM call to check the first one, ask whether a schema-constrained response removes the need for the check entirely. It often does.
A deterministic pass, even a partial one, before any LLM call is the cheapest tier you have. It will not handle every case. Every case it does handle is a call you never had to make, pay for, or wait on.
Give the deterministic tier a way to prove it is right. Arithmetic and cross-field checks, such as Billed = Allowed + Not Covered, are what let a non-AI path be trusted instead of merely fast.
Prompt length and model choice are independent levers. Optimize both, not whichever one you thought of first.
Log which tier resolved each request. You cannot tune a routing ladder you cannot observe.
Benchmark small, validate at scale, then continuously monitor after you ship. Watch cost, latency, and accuracy against the live system for drift, rather than treating launch as a one-time validation event. The last step is the one teams skip most often, and it is the one that tells you whether the first three worked.
Reflections
The pattern underneath all of this is not an AI pattern. It is the same instinct that says do not query the database when you can read the cache, and do not call the service when you already have the answer. LLMs made that instinct easy to forget, because the model is genuinely capable of doing every step, so it is tempting to let it.
Token usage is not fixed just because something is an AI feature. The most durable approach is designing the routing ladder up front rather than retroactively engineering cost down later. Treat the LLM as the fallback for genuinely hard cases, not as the default tool for every case. Then measure whether you were right.
