LLM API Relays and AI Agents: What a 20-Relay Audit Found About Tool Calls and Token Billing
A new preprint tested relays reselling GPT-6 Astra and Claude Opus 5. Four returned tool calls that the request had switched off.
Many teams do not call a model provider directly. They go through a relay: a middleman API that resells access to frontier models, often at a lower price, sometimes with an easier sign-up or a single key for several providers. For a chat app, the main question is whether the answers are good. For an AI agent that can call tools, the questions are sharper: does the relay respect the permissions you set, and do you pay for what you actually sent?
A preprint posted to arXiv on 5 November 2026, “What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays” by Xiaodong Wu, Zhimin Zhao, Qi Li, Chaklam Cheong, Xiangman Li and Jianbing Ni, puts numbers on both. Here is what it reports, what it does not claim, and what I would change in an agent stack because of it.
What the study tested
The authors tested 20 relays offering GPT-6 Astra and Claude Opus 5, and compared their answers with batch-matched responses from the official APIs. They also checked tool-permission behavior, reported token counts and, where account balances were visible, what was actually charged. It is a v1 preprint, not yet peer reviewed, and the authors themselves call for repeated measurements.
Finding 1: a disabled tool is a request, not a guarantee
In the first round, four relays returned tool calls even though the request instructed that tools were disabled.
For an agent, this is the finding that matters most. Many harnesses use the API’s own controls to scope a step: “this step only summarizes, no tools.” If the harness then executes whatever tool call comes back, the permission was enforced by the relay, not by you.
Finding 2: token counts that do not match the official API
For identical requests, five relays reported between 4,200 and 60,000 more input tokens than the official API. Where balances were available (six services), charges followed reported usage at listed prices, and at one service that included the extra reported input tokens.
So the billing math was consistent with the relay’s own counts. The problem is the counts themselves. An agent loop sends many requests per task, so a surplus on each call compounds into the per-task cost, and into any budget cap that trusts the provider’s usage numbers.
Finding 3: behavioral differences, not proof of substitution
On knowledge questions, the first-round analysis flagged 7 of 10 low-price relays and 2 of 10 same- or higher-price relays for behavioral differences from the official responses. On a retest of 35 pairs, seven gained or lost a flag under the original rules. The paper’s conclusion is careful: advertised models and prices are not enough to judge a relay’s reliability. A flag is a difference, not evidence that a different model was served.
What it did not find
No injection was confirmed in 3,332 screened responses to a fixed, non-adversarial code and setup workload. That is a useful result for that workload. It is not a finding that relays never alter responses.
A concrete example
Picture a customer-support agent with two tools: lookup_order and issue_refund. The plan has a step that summarizes a long ticket thread before the agent decides anything. To keep that step safe and cheap, the code sends it with tools disabled.
Get Kiran Yenaganti’s stories in your inbox
Join Medium for free to get updates from this writer.
Now route that call through a relay that returns a tool call anyway. If the harness treats any returned tool call as something to execute, a summarization step can issue a refund. No prompt injection is needed, and no one attacked anything. The control lived in a request flag, and the flag was not honored.
The same step also shows the cost side. A long thread is exactly where a relay that reports tens of thousands of extra input tokens turns a cheap summary into an expensive one, and a per-call budget built on reported usage stops the agent early or lets it overspend without anyone noticing.
What I would change in an agent stack
- Enforce tool scope in your own runtime. Keep an allowlist per step and reject any tool call outside it before execution, whatever the API flag said. Treat an unexpected tool call as an event worth logging.
- Count tokens yourself. Tokenize what you send locally, compare it with the provider’s reported usage, and alert when they diverge beyond a set margin.
- Set budgets per task, not per call. A spend cap on the whole task, checked outside the model, catches compounding surpluses that per-call limits miss.
- Record the route in the audit trail. Every action record should say which endpoint served the model call that led to it, so a strange decision can be traced back to the path it took.
- Re-test on a schedule. The retest in the paper changed flags for several pairs. One clean check is not a property of the service.
The broader point
A relay is part of your trust boundary. Anything you configure in the request, from disabled tools to output limits, is enforced by whoever answers that request. Runtime governance means the checks that matter run where you control them: before a tool executes, against the task’s budget, and in records you own.
About my work
I research runtime governance for autonomous AI agents: evidence checks before action, policy enforcement at execution time, risk-based human escalation and audit records. Notes, case studies and an interactive demo are at kiranyenaganti.com.
Written with AI assistance from my research notes.
