Why Price Per 1M Tokens Is a Misleading Metric for LLM Costs
TL;DR
Comparing LLMs by token pricing alone can lead you to choose worse, more expensive models. Cost per task tells the real story.
Last updated: July 9, 2026
Every AI pricing page leads with the same number: dollars per million tokens. OpenAI, Anthropic, Google, DeepSeek - they all compete on this metric. But comparing LLMs by their per-token pricing alone is fundamentally flawed. For a full breakdown of what those headline numbers actually look like across providers, see our AI coding tools pricing guide.
A new analysis making the rounds on Hacker News breaks down exactly why - and proposes a better metric that changes which models look like good value.
The Core Problem#
Token pricing fails for two reasons that compound on each other:
Tokenizers are not standardized. Different labs use proprietary tokenizers that split identical text differently. The same content might require 160 tokens for GPT-4o but 200 tokens for GPT-4. Anthropic recently modified its tokenizer, causing a 30% increase in tokens for the same input.
When you compare $X per million tokens across providers, you are comparing apples to oranges. A "token" from OpenAI is not the same unit as a "token" from Anthropic.
Token efficiency varies dramatically. Hidden chain-of-thought processing - where models reason before producing output - consumes tokens billed at standard rates but varies wildly between models and use cases. A model that thinks more might produce fewer output tokens but consume many more thinking tokens you do not see in the final response.
Cost Per Task: A Better Metric#
The proposed alternative: measure "cost per benchmark task" using real benchmark data. This reveals actual economic value delivered rather than nominal pricing.
The comparison table from the original analysis demonstrates the problem starkly:
- GPT-5.5 costs more per token than Claude Opus 4.8, yet completes tasks at nearly half the price
- DeepSeek V4 Pro charges dramatically less per token ($0.435/$0.87 input/output) but costs only $0.04 to $0.05 per task - revealing extreme efficiency
- Claude Sonnet 5 underperforms Opus while costing more per task
That last point is notable. Anthropic's own initial benchmarks showed Sonnet 5 with lower performance at higher costs than expected. The per-token price looked competitive; the per-task economics did not.
What HN Is Saying#
The Hacker News discussion added several important nuances:
Caching matters enormously. One commenter noted: "Caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens)." A model with better caching support can be dramatically cheaper in multi-turn conversations even with higher nominal token prices.
Thinking levels change the equation. "Setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash)." More thinking can mean fewer failed attempts, fewer tokens wasted on wrong paths, and faster completion.
Benchmark difficulty matters. Cost per benchmark task is only useful if the benchmark matches your workload. "Cost per benchmark task is meaningless if your task is difficult enough that the cheaper model has no chance of cracking it." For trivial tasks, the smaller model wastes tokens backtracking while the larger model does it right the first time.
Local LLM users see this too. "tok/s isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals in the minimum amount of time." Some models have better tok/s but are so verbose they generate many more tokens - making clock time longer despite the higher throughput.
The real problem is black box uncertainty. "You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume 'the cheaper and older model is probably going to cost less overall.'" This assumption is often wrong.
The Subscription Wrinkle#
Several commenters pointed out that token pricing is even more misleading for subscription users. Monthly plans price tokens extremely differently than their per-token billing rates. Most developers using Claude Code or ChatGPT Plus are not paying API rates at all.
Cost-per-task analysis should ideally account for subscription token allocations, but that data is rarely available.
Practical Implications#
If you are selecting models based on per-token pricing alone, you are likely choosing suboptimal solutions. Here is what to do instead:
Run your own benchmarks. The only cost metric that matters is cost for your actual workload. Generic benchmarks help, but your task distribution is unique. A tool like OpenRouter makes it easy to swap models and compare cost-per-task across providers without rewriting your integration each time.
Track total cost per task. Instrument your agent workflows to log total tokens consumed (input, output, thinking) and correlate with task success rates. A model that fails 20% of the time costs more than one that succeeds consistently even at higher per-token rates.
Account for caching. Multi-turn conversations with good cache hit rates can reduce costs 10x. Check each provider's caching behavior with your prompt patterns.
Test thinking levels. Higher thinking settings sometimes complete tasks faster and cheaper by avoiding failed attempts. Do not assume "low" is always cheapest.
Consider latency. A model that costs more per token but finishes in 2 seconds might be cheaper than one that takes 30 seconds if your time has value. One commenter wanted a model for commit messages that finishes quickly - high benchmark scores were irrelevant if it took a minute.
The Open Model Question#
Several commenters advocated for local models to avoid per-token uncertainty entirely. Fixed hardware costs are predictable; token costs are not. Tools like Ollama and LM Studio make running that hardware-bound floor straightforward if you want to test the tradeoff yourself; see our roundup of the best local coding LLMs for current options.
The counterargument: open models are not yet competitive for end-to-end agentic workflows. They excel at bounded tasks but struggle with the kind of multi-step reasoning that frontier models handle.
One detailed response described success with Mimo v2.5 at $0.017 per million tokens - building an orchestrator that handled planning, execution, and review with quality "that makes me laugh at things like Opus." The open model space is catching up fast.
The Bottom Line#
Price per million tokens is a unit measure, not a value measure. It tells you what you pay for a unit of computation but says nothing about what that computation accomplishes.
Just as price per gallon does not tell you trip cost without knowing fuel efficiency and distance, price per token does not tell you task cost without knowing model efficiency and task complexity.
The right question is not "which model is cheapest per token" but "which model completes my tasks most cost-effectively." Those are often different answers.
FAQ#
Why is price per million tokens misleading for comparing LLMs?#
Tokenizers are not standardized, so the same text splits into different token counts across labs, and token efficiency varies widely because hidden chain-of-thought tokens are billed at standard rates. A token from one provider is not the same unit as a token from another.
What is cost per task?#
Cost per task measures what a model actually costs to complete a benchmark task, using real benchmark data. It shows economic value delivered rather than nominal pricing - for example, GPT-5.5 costs more per token than Claude Opus 4.8 yet completes tasks at nearly half the price.
How should I measure LLM costs for my own workload?#
Run your own benchmarks, log total tokens consumed (input, output, thinking) per task and correlate them with success rates, check each provider's caching behavior with your prompt patterns, test different thinking levels, and factor in latency.
Does a higher thinking setting always cost more?#
Not necessarily. One HN commenter found that setting thinking to high instead of low made tasks complete faster and cheaper with Gemini 3.0 Flash, because more thinking can mean fewer failed attempts and fewer tokens wasted on wrong paths.
Continue Reading#
Sources#
Get the next comparison like this in your inbox
One email a week on AI and the rest of the AI dev stack. Free.
Read next
LLM Architectures Got Complicated Fast
LLM architectures got complicated: MoE routing, mixed attention variants and integrated vision encoders replaced the simple transformer. What changed and why.
6 min readMozilla's State of Open Source AI Report: The Gap Is 3%, But Deployment Remains the Real Problem
Mozilla's State of Open Source AI report puts open models within 3.3% of closed ones, yet only 51% reach production. The deployment barriers explained.
8 min readPrompt Injection is Role Confusion - New ICML Research Explains Why LLMs Can't Tell Friend from Foe
Prompt injection is role confusion, per an ICML 2026 paper: LLMs judge who is speaking by style, not role tags, so text that sounds like the system wins.
7 min readNew here? Start with
Technical content at the intersection of AI and development. Building with AI agents, Claude Code, and modern dev tools - then showing you exactly how it works.
