Airbnb Raced a Fine-Tuned 2B Model Against GPT-6.1 Sol, Claude Sonnet 5.5, and Jev. The 2B Model Was the Only One That Hit Both Targets.
A production benchmark on one real classifier, what “max F1” hides, and the request rate at which self-hosting stops paying off
Airbnb fine-tuned a 2-billion-effective-parameter Gemma model for a production content classifier. Its max F1 went from 0.563 to 0.902, on par with the best frontier models Airbnb tested. Median latency was 35 milliseconds. GPT-6.1 Sol and Claude Sonnet 5.5 took about 900 milliseconds at the median and several seconds at p99, against a latency budget of 200.
That result alone would be a familiar “small models are underrated” story. The post by Airbnb engineers Clay Gibson and Adel Elmalaha is more useful than that because it puts real numbers on the boundaries. It shows where prompting stops helping, why a probability beats a yes/no answer, and the request rate below which the frontier API is the cheaper option. It’s also the first production comparison I’ve seen that puts TypeSafe’s Jev decision model next to the generative frontier models.
The post doesn’t give a publication date. Its images are dated November 2026, and its pricing uses October 2026 list rates.
The task: a gate, not a chat
The classifier screens content for violations of Airbnb’s pricing and payment transparency policy. It runs before content is published, so it’s in the critical path. It’s called thousands of times per minute and has 200 ms to return a score. A false positive blocks legitimate content, so precision matters more than in a typical moderation queue.
The setup:
- Evaluation set: 10,000 held-out examples, weighted to correct for how production traffic was sampled. A subset of models was re-checked on 184,000 examples, and their max F1 stayed within 0.01 of the reported numbers.
- Fine-tuned model: Gemma4-E2B, trained on hundreds of thousands of de-identified in-domain examples from users who hadn’t opted out of AI training. Training ran through GLOWC, Airbnb’s internal framework for turning open-weight language and vision-language models into classifiers. GLOWC also provides the shared batch-inference and serving infrastructure.
- Serving: every open-weight model ran on vLLM 0.28.0 on a single NVIDIA RTX Pro 6000. Models up to 12B parameters ran in bfloat16, and larger models used public FP8 or NVFP4 checkpoints.
- Prompt: unless noted, every model got the same concise prompt of about 400 words.
- Proprietary models: called through vendor APIs in low-thinking mode. Airbnb found that more thinking didn’t meaningfully improve classification.
A note on the “2B.” Google’s model card lists Gemma 4 E2B at 2.3 billion effective parameters and 5.1 billion in total. The gap comes from Per-Layer Embeddings, which give each of the model’s 35 layers its own small per-token embedding table. Those tables count toward the file size but not toward the “effective” parameters Google uses to describe the model’s per-token compute. Airbnb describes it as about 2B effective parameters.
The concept: a verdict is a point, a score is a curve
This is the part of the post I’d most want an engineer to take away, because it isn’t really about model size.
Most generative models answer a classification prompt with a verdict: “violation” or “not a violation.” A verdict gives you exactly one precision/recall operating point. With the concise prompt, GPT-6.1 Sol landed at 0.82 precision and 0.96 recall. If the business decides false positives are too costly and wants precision at 0.95, the only lever is rewriting the prompt and re-running the evaluation.
A model that outputs a probability works differently. You choose the threshold. Raise it and precision goes up while recall drops; lower it and the opposite happens. Airbnb’s fine-tuned E2B produces a score, and so does Jev, which TypeSafe designed to return a calibrated decision rather than prose. That’s why Airbnb reports max F1 as its headline metric:
- For models with a score, they sweep the threshold and report the best F1.
- For verdict-only models, they report the F1 of the single verdict.
The team normally prefers PR-AUC for imbalanced data like this, but verdict-only models have no curve to integrate. They say PR-AUC and accuracy rank the models almost identically.
The arithmetic is quick to check. Sol’s single operating point works out to F1 ≈ 0.884, a bit below E2B’s 0.902. The bigger difference is that E2B can be moved along its curve to wherever the policy team wants, and Sol can’t. Here is a small script that computes Sol’s F1 and sweeps a threshold on toy data so you can see the curve:
"""Part A of classifier_math.py: a verdict is one point; a score is a curve."""
import random
def f1(precision, recall):
return 2 * precision * recall / (precision + recall)
sol_f1 = f1(0.82, 0.96)
print(f"Verdict-only model at P=0.82, R=0.96 -> F1 = {sol_f1:.3f}")
random.seed(7)
data = [(1, random.gauss(0.72, 0.15)) for _ in range(1_000)] + \
[(0, random.gauss(0.30, 0.15)) for _ in range(9_000)]
def pr_at(threshold):
tp = sum(1 for y, s in data if y == 1 and s >= threshold)
fp = sum(1 for y, s in data if y == 0 and s >= threshold)
fn = sum(1 for y, s in data if y == 1 and s < threshold)
p = tp / (tp + fp) if tp + fp else 1.0
r = tp / (tp + fn)
return p, r
best = (0.0, None)
for t in [0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
p, r = pr_at(t)
score = f1(p, r) if p + r else 0.0
best = max(best, (score, t))
print(f"threshold {t:.2f}: precision {p:.3f} recall {r:.3f} F1 {score:.3f}")
print(f"max F1 = {best[0]:.3f} at threshold {best[1]:.2f}")On the toy data (10% positives, overlapping score distributions), F1 peaks around 0.81 at threshold 0.60. At 0.70, precision rises to 0.95 and recall falls to 0.59. A scored model lets you choose that trade explicitly; a verdict-only model makes it for you.
The results
Airbnb plotted everything on a quality-versus-latency chart. The post only states exact values for some models, so I’m reporting only those.
Fine-tuned Gemma4-E2B. Max F1 went from 0.563 before fine-tuning to 0.902 after, on par with the best proprietary models. Latency was 35 ms median and about 85 ms p99. The authors say no other model came close on both quality and latency at once.
GPT-6.1 Sol and Claude Sonnet 5.5. Quality was at the top of the chart. Latency was about 900 ms median, with multi-second p99s, well outside the 200 ms budget.
GPT-6 Luna. The fastest proprietary generative model Airbnb tested, but it was still slower and less accurate than the open-weight Gemma4–31B running on one GPU.
Jev. The most accurate off-the-shelf model Airbnb tested under 100 ms. Because it returns a probability, it can be thresholded like the fine-tuned model. The post doesn’t give its F1. For readers of my earlier Jev pieces, this is the first production benchmark I’ve seen where a decision model sits next to the generative frontier models. Here it beat Luna at about the same cost, but it didn’t reach the fine-tuned specialist.
Get Varun Nuthalapati’s stories in your inbox
Join Medium for free to get updates from this writer.
Off-the-shelf open-weight models. Many fit within the latency budget, but their quality plateaued well below both the best proprietary models and the fine-tuned classifier.
One footnote is worth repeating for anyone serving newer architectures. Latency didn’t track parameter count. Airbnb notes that vLLM’s current kernels are less optimized for hybrid linear-attention layers than for standard or sliding-window attention, so two models of similar size can have quite different latencies.
The prompt-engineering ceiling
The obvious objection is that a better prompt would close the gap. Airbnb tested it with a detailed 2,700-word prompt that added worked examples and common false positives.
- Proprietary generative models gained one to three F1 points, which brought them roughly level with the fine-tuned classifier. Their latency stayed far outside the budget.
- Models that already met the latency budget gained only modestly, and a large gap remained.
That split is the core argument. Prompting improves the models that are already too slow and does little for the models fast enough to deploy. The authors open the post with a peanut-butter-and-jelly-sandwich analogy to make the same point: there’s a limit to how much better instructions can do. In their list of when to fine-tune, they put it plainly: hundreds of thousands of labeled examples taught distinctions that the 2,700-word prompt couldn’t.
The cost math, including the break-even point
Airbnb expresses every cost as a multiple of one self-hosted GPU, priced at public cloud rental rates for an RTX Pro 6000. The post doesn’t give the hourly rate. Proprietary costs come from the tokens actually used during evaluation, priced at October 2026 list rates without negotiated discounts.
- Frontier models, concise prompt: about two orders of magnitude more than one GPU at a load of thousands of RPM. The detailed prompt added another 20–40%.
- Lower-cost proprietary options: about 5× the GPU cost, with lower accuracy than the fine-tuned model.
- Luna vs. Jev: similar cost on this task. With the detailed prompt, Jev’s cost quadrupled, because its API has no cached-input discount.
- Break-even: self-hosting becomes cheaper above roughly 30 RPM. Below about 15–30 RPM, Claude Sonnet 5.5 and GPT-6.1 Sol cost less than a dedicated GPU.
The comparison is generous to the APIs. Airbnb assumed a 71% cache-hit rate on cached input, which it says is higher than it actually observed.
The two headline numbers are consistent with each other. A GPU costs the same whether it handles 30 or 3,000 requests per minute, until you need a second one. An API bill grows linearly with traffic. If the two break even at 30 RPM, then at R RPM the API costs R/30 times the GPU:
"""Part B of classifier_math.py: what the break-even point implies."""
G = 1.0 # one GPU's monthly cost, as the unit
BREAK_EVEN_RPM = 30 # Airbnb: self-hosting wins above ~30 RPM
k = G / BREAK_EVEN_RPM # API cost per unit of RPM
for rpm in [15, 30, 300, 1_000, 3_000, 6_000]:
print(f"{rpm:>5} RPM -> API costs {k * rpm:6.1f}x the GPU")
for uplift in (0.20, 0.40): # the detailed prompt's extra cost
print(f"+{uplift:.0%} prompt cost -> break-even falls to "
f"{BREAK_EVEN_RPM / (1 + uplift):.1f} RPM")15 RPM -> API costs 0.5x the GPU
30 RPM -> API costs 1.0x the GPU
300 RPM -> API costs 10.0x the GPU
1000 RPM -> API costs 33.3x the GPU
3000 RPM -> API costs 100.0x the GPU
6000 RPM -> API costs 200.0x the GPU
+20% prompt cost -> break-even falls to 25.0 RPM
+40% prompt cost -> break-even falls to 21.4 RPMA 100× ratio lands at about 3,000 RPM, which matches Airbnb’s “thousands of requests per minute.” This is a simplified model that ignores engineering time and the cost of fine-tuning, but it shows why the result is so lopsided. A self-hosted GPU is a fixed cost, so once one GPU can carry the traffic, every extra request is essentially free. With an API, every extra request costs money. Better prompts that add tokens make this worse by lowering the break-even point.
When to fine-tune, per Airbnb
The authors give four conditions that should all hold:
- High volume. Above roughly 30 RPM for this task, one GPU is cheaper than frontier APIs.
- Tight latency. E2B returned scores at about 85 ms p99, while frontier APIs ran about 900 ms at the median. Faster API tiers exist, but at a premium the post doesn’t quantify.
- A need for a threshold. Scores let you move the precision/recall trade-off without rewriting prompts.
- Labeled data. Hundreds of thousands of in-domain examples taught distinctions that a 2,700-word prompt couldn’t.
They’re also explicit about the opposite case. For low-volume or frequently changing tasks, a frontier model with a good prompt can be the better choice. The results come from a single classifier, though the authors say they saw similar trade-offs with other models they fine-tuned.
The takeaway
This is the third company post I’ve covered recently that reaches a similar conclusion, though by different routes. Netflix trained a recommender with a scoring head that never generates text. Pinterest distilled a frontier teacher into a 4B student. Airbnb fine-tuned a 2B-effective model and found that the latency budget, more than the leaderboard, determined which model it could use.
The general lesson is that what you need from the model’s output determines which model fits. A gate that has to answer in 200 ms with an adjustable threshold, thousands of times a minute, needs a score, low latency, and a fixed cost. Frontier models are the right tool for writing the labels, handling low-volume tasks, and covering cases nobody has labeled yet. For high-volume classification in production, Airbnb’s numbers favor a small fine-tuned model running on one GPU.
Appendix: the full script
The two snippets above are excerpts. This is the complete classifier_math.py in one block, so you can copy it into a file and run it with python3 classifier_math.py. It's plain Python with no dependencies.
"""Two pieces of arithmetic behind Airbnb's classifier benchmark.
Part A: why a score you can threshold beats a fixed verdict.
Part B: what Airbnb's break-even and cost-ratio figures imply about each other.
Pure Python, no dependencies. Run: python3 classifier_math.py
"""
import random
def f1(precision, recall):
return 2 * precision * recall / (precision + recall)
# ---------------------------------------------------------------------------
# Part A: a verdict is one point; a score is a whole curve
# ---------------------------------------------------------------------------
# Airbnb reports GPT-6.1 Sol (concise prompt) at precision 0.82, recall 0.96:
# a single operating point. Its F1 is fixed unless you rewrite the prompt.
sol_f1 = f1(0.82, 0.96)
print(f"Part A: verdict-only model at P=0.82, R=0.96 -> F1 = {sol_f1:.3f}")
print(" (Airbnb's fine-tuned Gemma4-E2B reported max F1 = 0.902)\n")
# A toy scored classifier: 10% positives, scores drawn from overlapping
# distributions. Sweep the threshold and watch precision/recall trade off.
random.seed(7)
data = [(1, random.gauss(0.72, 0.15)) for _ in range(1_000)] + \
[(0, random.gauss(0.30, 0.15)) for _ in range(9_000)]
def pr_at(threshold):
tp = sum(1 for y, s in data if y == 1 and s >= threshold)
fp = sum(1 for y, s in data if y == 0 and s >= threshold)
fn = sum(1 for y, s in data if y == 1 and s < threshold)
p = tp / (tp + fp) if tp + fp else 1.0
r = tp / (tp + fn)
return p, r
print(" threshold precision recall F1")
best = (0.0, None)
for t in [0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70]:
p, r = pr_at(t)
score = f1(p, r) if p + r else 0.0
best = max(best, (score, t))
print(f" {t:9.2f} {p:9.3f} {r:6.3f} {score:.3f}")
print(f" max F1 on this toy data = {best[0]:.3f} at threshold {best[1]:.2f}")
print(" A precision-first policy can pick a higher threshold instead;")
print(" a verdict-only model has no dial to turn.\n")
# ---------------------------------------------------------------------------
# Part B: break-even arithmetic
# ---------------------------------------------------------------------------
# Model: one self-hosted GPU costs G per month no matter the load (until you
# need a second worker). A pay-per-token API costs k per request, so its
# monthly bill scales linearly with requests per minute (RPM).
# Airbnb: self-hosting wins above ~30 RPM. That pins k: k * 30 = G.
G = 1.0 # express everything in "GPUs' worth" of cost
BREAK_EVEN_RPM = 30
k = G / BREAK_EVEN_RPM
print("Part B: frontier-API cost as a multiple of one GPU, if break-even is 30 RPM")
for rpm in [15, 30, 300, 1_000, 3_000, 6_000]:
print(f" {rpm:>5} RPM -> API costs {k * rpm:6.1f}x the GPU")
print(" ~100x ('two orders of magnitude') lands at ~3,000 RPM,")
print(" matching Airbnb's 'thousands of RPM' description of the workload.\n")
# The detailed 2,700-word prompt added 20-40% to frontier cost.
for uplift in (0.20, 0.40):
print(f" +{uplift:.0%} prompt cost -> break-even falls to "
f"{BREAK_EVEN_RPM / (1 + uplift):.1f} RPM")
Sources: — Clay Gibson and Adel Elmalaha, “Frontier quality at 1% of the cost: a production benchmark of LLM classifiers,” Airbnb Tech Blog (undated; assets and pricing from October–November 2026) — Google, Gemma 4 model card (ai.google.dev): E2B effective and total parameters, layer count, Per-Layer Embeddings — classifier_math.py: my own arithmetic on Airbnb's published figures, given in full in the appendix and run with its output shown above
