Jev vs LLMs on 24 Support Emails. Here’s What Happened (Part 2)

Part 1 explained what TypeSafe AI’s Jev is and what a calibrated decision means.

https://medium.com/@manucet439/jev-vs-llms-what-a-calibrated-decision-is-and-why-it-matters-part-1-2b82d238f6d5

This part puts it to the test: the same 24 customer support emails, triaged by an LLM and by Jev, measured on speed, cost, accuracy and, above all, whether each model’s confidence can be trusted.

• Speed and cost: Jev was 5–7× faster and about 2× cheaper than OpenAI’s cheapest model, gpt-6-luna, in two runs.
• Accuracy: comparable. Neither model was clearly better at getting answers right.
• Confidence: in run 1, the LLM routed 2 emails to the wrong team while 95–98% sure. Jev made no confident mistakes, and it flagged exactly the emails that were ambiguous.
• The surprise: Jev’s low confidence was pointing at a gap in our routing policy. Once we wrote the policy down, both models reached 100% on routing, Jev’s confidence on those emails rose to 1.00, and the LLM’s answers to questions we had not touched changed between runs.

This article combines my own experiment with publicly available material on Jev and calibration, listed under References.

The experiment: customer support email triage

We give both models the same job a support platform does thousands of times a day. For every incoming email, decide:

1. Team (Choice): billing, technical, sales or other
2. Refund request? (Noul): yes or no, as a probability
3. Urgency (Score): low, medium or critical

This is a good test because everyone understands it, the answers are checkable, and the cost of a wrong answer is easy to explain: an auto-approved refund that was not a refund costs money; a critical outage marked “low” costs a customer.

The dataset (ticket.csv) has 24 hand-labelled emails. Most are clear-cut. Some are deliberately ambiguous, because that is where calibration matters:

• “We were promised SSO in the sales call but it is not working on our plan. If this is not fixed we expect a refund.” Technical or billing? Refund or a threat?
• “After your price increase our renewal went up 40 percent… we want the old price honoured or a partial refund.”
• “Your competitor offers this for half the price. Convince me to stay.”

A good decision system should be sure about the easy ones and visibly less sure about these.

What we measure for each model: response time per email, input and output tokens, `estimated cost, accuracy on all three decisions, and calibration of the team decision. For the LLM we use the most common pattern in production today: Structured Outputs plus a self-reported confidence field.

The fair-comparison choices:

• LLM: gpt-6-luna, OpenAI's model for cost-sensitive, high-volume work, with reasoning effort set to none so it is as fast as it can be. A bigger model or reasoning would be slower and costlier, so this is the LLM's best case on speed.
• Jev: jev-1.13.0, pinned so results are reproducible.
• One warm-up call each before timing, so connection setup is not counted.
• Same email text and the same label definitions for both.

Python implementation

The full script, jev_vs_llm.py (about 390 lines), and the dataset, tickets.csv, are on GitHub: [https://github.com/manucet439/LLM_vs_Jev]. Below are the parts that matter. The code was tested with openai 3.33.0 and typesafe-sdk 0.7.5 in October 2026.

Setup

pip install -U openai typesafe-sdk pydantic python-dotenv

# .env
OPENAI_API_KEY=sk-...
TYPESAFE_API_KEY=ts-...

Shared label definitions

Both models get exactly the same definitions. The routing rule in the comment was added after run 1; why is explained below.

TEAM_CRITERIA = {
    # Routing policy: a refund request CAUSED by a product failure goes to technical first.
    "billing": "Charges, invoices, refunds, plan changes, payment methods, tax forms. "
               "Not refunds asked for because the product is broken (those go to technical)",
    "technical": "Something in the product is broken, unavailable, or the user needs help using it, "
                 "including when the customer also asks for a refund because of that failure",
    "sales": "Pricing questions before buying, quotes, discounts, upgrades, competitor comparisons",
    "other": "Feedback, praise, account deletion, or anything that fits no other team",
}
URGENCY_LEVELS = [
    "Low: a question or request that can wait a day or more",
    "Medium: the customer is blocked or unhappy and expects a reply today",
    "Critical: an outage, a deadline within hours, a security or privacy incident, or a threat to cancel",
]

Write the routing policy into the criteria. The first version of this experiment left “broken product plus a refund request” undefined, and that one gap produced every routing disagreement between the two models (see Run 1 Results).

1. The LLM way: generate JSON, ask how sure it is

We use the Responses API with Structured Outputs, so the JSON is guaranteed to match the schema. Confidence is a field the model fills in itself.

from typing import Literal
from openai import OpenAI
from pydantic import BaseModel, Field

class Triage(BaseModel):
    team: Literal["billing", "technical", "sales", "other"]
    team_confidence: int = Field(ge=0, le=100, description="How confident you are that `team` is correct, 0-100")
    refund_probability: int = Field(ge=0, le=100, description="Probability 0-100 that the email asks for a refund")
    urgency: Literal["low", "medium", "critical"]
client = OpenAI()
start = time.perf_counter()
resp = client.responses.parse(
    model="gpt-6-luna",
    instructions=LLM_INSTRUCTIONS,      # the label definitions above, as text
    input=email,
    text_format=Triage,
    reasoning={"effort": "none"},       # fastest setting: no hidden chain of thought
)
latency = time.perf_counter() - start
answer = resp.output_parsed             # Triage(team='billing', team_confidence=95, ...)
print(resp.usage.input_tokens, resp.usage.output_tokens)
print(resp.usage.output_tokens_details.reasoning_tokens)   # 0 with effort=none

2. The Jev way: typed questions in, probabilities out

One call answers all three questions in parallel. The backticks around `email` point each question at that field of the state, as TypeSafe's docs recommend.

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

jev = TypeSafeClient(model="jev-1.13.0")  
start = time.perf_counter()
resp = jev.system_one(
    state={"email": email},
    questions={
        "team": Choice(instructions="Which team should handle `email`?", criteria=TEAM_CRITERIA),
        "refund": Noul(instructions="Does `email` ask for money back, a refund, "
                                    "a reversed charge, or a partial refund?"),
        "urgency": Score(instructions="How urgently must we respond to `email`?",
                         criteria=URGENCY_LEVELS),
    },
)
latency = time.perf_counter() - start
team = resp.choices["team"]
print(team.choice, team.probabilities)     # billing {'billing': 0.94, 'technical': 0.03, ...}
print(resp.nouls["refund"].noul)            # 0.97
print(resp.scores["urgency"].score)         # e.g. 1.1, between "medium" and "critical"
print(resp.usage.input_tokens, resp.usage.output_tokens)

3. Measuring calibration

For each email we record the probability the model gave its chosen team and whether that team was correct. Then we bucket and compare, exactly as Guo et al. (2017) define it.

BINS = [(0.0, 0.5), (0.5, 0.7), (0.7, 0.9), (0.9, 1.01)]

def calibration(probs, correct):
    rows, ece, n = [], 0.0, len(probs)
    for lo, hi in BINS:
        idx = [i for i, p in enumerate(probs) if lo <= p < hi]
        if not idx:
            continue
        avg_p = mean(probs[i] for i in idx)          # what the model said
        acc = mean(correct[i] for i in idx)          # what actually happened
        ece += len(idx) / n * abs(avg_p - acc)       # weighted gap
        rows.append((lo, hi, len(idx), avg_p, acc))
    return rows, ece

The script also simulates a refund automation rule on both models: auto-refund when p ≥ 0.9, ignore when p < 0.1, send the rest to a human. It counts how many emails each model automates and how many automated decisions were wrong.

Run 1: an incomplete policy (like most real ones)

Run 1 used team definitions without the routing rule shown above, so nobody had said where “broken product plus a refund request” should go. Real routing policies almost always have gaps like this. (Two urgency labels were corrected after the run; the numbers below reflect that.)

Where the two models disagreed

On 20 of the 24 emails both models chose the same, correct team with high confidence. Every difference came from four emails that are genuinely ambiguous:

All three LLM mistakes follow one pattern: the product is broken and the customer asks for money back, and the LLM routed on the word “refund”. Jev got two of the three right. More importantly, it was below 0.9 on all four, so it signalled that these needed a second look. The LLM was 95–98% sure on two of its wrong answers.

What a 90% automation rule would do

Rule: route automatically if p ≥ 0.9, otherwise a person checks

Jev asks a person to look at 3 more emails, and in exchange no wrong routing goes through unnoticed. That trade-off is the business value of calibration.

Get Manu Madhavan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Calibration in numbers: the LLM put 23 of 24 emails at 0.9 or above and was right on 91% of them, so its confidence could not separate its 2 confident mistakes from its correct answers. Jev put 20 emails at 0.9 or above, all correct, and its one mistake sat at 0.71. Calibration error (ECE): LLM 0.090, Jev 0.062.

Also worth noting:

• Refunds: both models caught all 7 refund requests. The LLM’s refund probabilities were only ever 0, 0.95 or 1.0; Jev’s spread from 0.01 to 0.99 and sent one borderline email (#20, “card declined but pending”, at 0.10) to a person.
• Tokens: Jev used about 205 more input tokens per email than the LLM, a fixed overhead from how questions are packaged. It was still cheaper because its input price is lower and its output is free. Against gpt-6.1-sol the same LLM tokens would cost about $930 per million emails, roughly 40× Jev.
• Latency: Jev’s calls clustered around 300 and 400 ms, which looks like network round trips between India and the US rather than model time.

Run 2: after clarifying the policy

Run 1 exposed a gap in our own instructions: nobody had said where “broken product plus a refund request” should go. For run 2 (21 October 2026) I added the rule shown in the code above, a refund caused by a product failure goes to technical, and ran both models again. Nothing else changed: same emails, same refund question, same urgency definitions.

Input tokens rose by about 30 per email for both models because the team definitions got longer. Output tokens were exactly the same as in run 1 for both, which confirms that each model’s output has a fixed shape. Jev was also more consistent on speed: its p95 was 32 ms above its median, against 640 ms for the LLM.

Finding 1: Jev’s uncertainty was pointing at our policy

Once the policy was written down, both models routed every email correctly. Jev’s confidence on the three policy-dependent emails rose from 0.51–0.88 to 1.00. Its low confidence in run 1 was not a weakness of the model; it was correctly reporting that our instructions were ambiguous. The LLM was 95–98% sure in both runs, whether it was wrong (run 1) or right (run 2), so its number told us nothing about the gap.

Finding 2: the LLM made a confident refund mistake

Email #17, “…If this is not fixed we expect a refund”:

• LLM: refund probability 0.95 in run 1, 0.00 in run 2, so the refund rule ignored it.
• Jev: 0.96 in both runs.

Changed only the team definitions, yet the LLM’s answer to the refund question flipped from almost certainly yes to certainly no. The likely reason is that the LLM answers all three questions inside one generated JSON, so editing one definition shifts the others. Jev answers each question independently. (#17 is a conditional refund, so its label is debatable, but the same email moving from 0.95 to 0.00 with full confidence is the problem either way.)

Finding 3: Jev’s other answers did not move; the LLM’s did

• Jev: every refund and urgency answer was identical to run 1. Only the team probabilities moved, and those were the definitions we edited.
• LLM: three urgency answers changed to “low” although the urgency definitions were untouched, and all three are now wrong (#12 “I’d like a refund”, #20 “card declined but pending”, #23 “delete my account”). That is why its urgency accuracy fell from 92% to 79%.

Calibration itself now looks perfect for both models (ECE near zero), because there were no routing mistakes left to measure. Calibration only shows its value where there are mistakes or genuine ambiguity; on easy data every model looks calibrated. Run 1 is the evidence; run 2 is the control.

Lessons for production

1. Use Jev-style models for the fast, high-volume decisions. Across two runs Jev was 5–7× faster and about 2× cheaper than OpenAI’s cheapest model, with comparable accuracy. Against larger LLMs the cost gap grows to tens of times.
2. Treat low confidence as a signal, not a failure. Jev’s uncertain answers pointed straight at a gap in our routing policy. Review the emails a calibrated model is unsure about; they often show where your rules are unclear.
3. Write the policy down before blaming the model. One sentence in the team definitions took both models to 100% routing accuracy.
4. Retest everything after any prompt change. With the LLM, editing one definition changed answers to two unrelated questions. Jev’s independent questions did not move.
5. Set thresholds from calibration, not from gut feel. A rule like “automate at 90%, else a person checks” only works if 90% really means 90%. Measure it on your own data.
6. Keep the LLM for what it does best. Jev decides; the LLM writes the reply, explains, and generates tool arguments.

Limits of this experiment

• Small dataset. 24 emails make a good live demo but not a production decision. Each email is worth about 4 percentage points, so most accuracy differences here are within noise, and several calibration bands hold only 1 or 2 emails.
• Two runs. The LLM’s answers changed between runs; more runs would show how much.
• Debatable labels. Some urgency labels (#4, #17, #23) and the conditional refund in #17 are judgement calls.
• One LLM. Only gpt-6-luna with reasoning off was tested. A larger model or reasoning might be more accurate, but slower and costlier. OpenAI's new Decisions API, which also returns probabilities, would be a natural third column.

References

1. Part 1 of this series: Jev vs LLMs: What a Calibrated Decision Is and Why It Matters.
2. TypeSafe AI. Confidence — probabilities, the confidence field, and thresholds.
3. Flavio Copes. How to use Jev with Python, September 2026 — SDK usage, pricing and rate limits.
4. DataCamp. Jev API Tutorial: Building a Ticket Router.
5. TechCrunch. Coverage of the Jev launch, September 2026 — including the Vercel results.