The Prototype Works. Now Comes the Part Nobody Budgeted For.
The demo is the easiest milestone in an AI feature’s life, and the most misleading. It runs on a personal key, with no ceiling, no attribution, and no log worth the name — and it works, which convinces everyone the feature is nearly done. Then production asks its four questions. Whose key is this? What stops it spending all weekend? What did that call cost? Where is the record when it fails? None of those questions is about the model. All of them are about configuration — and configuration is where the four weeks go.
Fund the organization, and let the ceiling bind early.
The first job is money, and the trick is ordering. Through a gateway, each call’s estimated cost is reserved against the organization’s balance before the request is sent. If the balance cannot cover the call, it comes back 402 and no provider is ever contacted. That ordering is the whole feature: the refusal happens before the spend exists. The demo pattern has no equivalent — a personal key spends until a human notices, usually on an invoice, usually at the worst possible moment.
One key per service, or attribution never happens.
The second job is identity. A production setup creates a scoped virtual key for every (environment, service) pair: a name, exactly one team, its own budget, allowed models and endpoints, an IP allowlist, an expiry — and a full value that is shown exactly once. The shortcut, one shared key for everything, feels pragmatic for about a week. Then finance asks which service spent what, and the answer is a shrug, because every call merged into a single undifferentiated line item. Attribution cannot be backfilled. The separation either exists at the key or it does not exist at all.
Get nRouter.ai’s stories in your inbox
Join Medium for free to get updates from this writer.
Change the base URL. Keep everything else.
The third job is the integration itself, and it is embarrassingly small: the API is OpenAI-compatible, so the client already in your codebase takes a new base URL and keeps its request and response types. What matters is the discipline from the first call. Persist the request ID header on success and failure — it is the thread that joins the response, the request log, and the spend record. Read the cost header when it is present. And when it is absent, record the cost as unknown. The single most corrupting habit in AI cost reporting is defaulting a missing cost to zero: the dashboards stay green, the totals stay plausible, and they are wrong.
Attach the budget where the risk lives — and make retries respect it.
The fourth job is the ceiling, and it has a contract your code can read. Budgets enforce in three modes — Block, Warn, Throttle — and Warn alone is a notification wearing a cap’s clothing. Scope decides the refusal: organization, team and user budgets return 402 with the code budget_exceeded; per-key budgets return 429 with key_budget_exceeded. That distinction is not trivia. A rate-limit 429 clears itself; a budget 429 does not clear until the window resets. Retry logic that branches on the status code alone will spend the night backing off against a refusal that time cannot fix. Branch on the code field, pair the budget with rate limits on the same key, and the ceiling becomes something your system understands instead of something it discovers.
Verify it fires. Then ship.
None of this counts until it has been watched working. The verification fits in ten minutes: the first response carries a request ID and a cost header; the call appears in the request logs, findable by that ID; the ledger moves by the settled cost; and a throwaway key with a tiny budget refuses its second call with the budget code. Delete the key. Configuration never watched firing is a hypothesis — and after the test, adding the second model is a string change, not a second integration project.
