The Economics of Thinking

The previous posts established that spending more inference compute buys accuracy. This post is about the price — the economics and trade-offs of test-time compute. Thinking costs money (tokens) and time (latency), and these costs scale with how much the model thinks. Understanding the trade-off — when extra thinking is worth it and when it isn’t — is what separates using reasoning models well from using them wastefully. This is the practical heart of the series.

Thinking tokens cost money

The most direct cost of test-time compute is tokens. Reasoning models generate a chain of thought before the answer, and that reasoning can be long — often many times the length of the final answer. Since inference is billed (and consumes compute) per token, you pay for the thinking:

• Reasoning tokens are billed. The model’s internal reasoning consumes output tokens just like the answer does, so a reasoning model that thinks extensively can cost several times more per query than a standard model answering directly — even for the same final answer length. The “hidden” thinking is a real, paid cost.
• Sampling and search multiply it. Inference-time techniques compound this: best-of-N generates N candidates (≈N× the tokens), search explores many branches (potentially far more). Spending compute for accuracy means spending tokens, and the multiplier can be large.
• Cost scales with difficulty and effort. Because harder problems trigger more thinking (and you may dial up reasoning effort), cost is variable per query — an easy question is cheap, a hard one with heavy reasoning is expensive. This variability is new: standard models have roughly predictable per-query cost; reasoning models’ cost depends on how hard they think.

The practical consequence: reasoning is a purchase, and you should treat it like one. For a high-value problem (a hard analysis, a critical decision, code that must be correct), paying for extensive thinking is easily worth it. For a simple, high-volume query, paying reasoning-model prices for thinking the problem doesn’t need is waste. The cost framing — you’re buying accuracy with tokens — should guide when you reach for reasoning.

Latency: thinking is slow

The second cost is latency. A model that thinks before answering is slower to respond — sometimes dramatically:

• Thinking delays the answer. Every reasoning token is generated sequentially before the answer begins, so a long chain of thought means the user waits longer for the first useful output. A reasoning model working hard can take many seconds (or more) to respond, versus a near-immediate answer from a standard model. (The LLM-serving series explains why generation is sequential and latency-bound.)
• It clashes with interactivity. For interactive uses (chat, autocomplete, anything where a human waits), high latency hurts the experience. Users tolerate a pause for a hard problem but not for a simple one. Latency-sensitive applications may not be able to afford much thinking.
• Sampling/search add wall-clock or infrastructure cost. Generating many candidates can be parallelized (trading money for speed) or done sequentially (trading speed for money) — either way, more compute means either higher latency or more hardware. There’s no free lunch.

Latency makes test-time compute a fit question, not just a cost question: some applications simply can’t wait for extensive reasoning, no matter the accuracy gain. Matching the amount of thinking to the latency budget is as important as matching it to the money budget.

When reasoning helps — and when it hurts

Crucially, more thinking is not always better. Test-time compute helps on some problems and is wasteful or even harmful on others, and knowing the difference is key:

• Reasoning helps on hard, multi-step problems. Math, complex coding, logic, planning, careful analysis — problems where working through steps genuinely improves the answer. Here, thinking pays for itself in accuracy. This is reasoning models’ home turf.
• Reasoning barely helps on simple problems. For factual lookups, simple classifications, straightforward conversational replies, or formatting tasks, there’s little to reason through — extended thinking adds cost and latency with negligible accuracy gain. Using a heavy reasoning model here is pure waste; a standard model is faster and cheaper.
• Overthinking can hurt. On easy problems, forcing a model to think extensively can even reduce quality — it may overcomplicate a simple answer, second-guess a correct initial response, or wander. More compute has diminishing and sometimes negative returns past the point the problem warrants. “Overthinking” is a real failure mode, not just inefficiency.
• Diminishing returns everywhere. Even on hard problems, accuracy gains from more thinking eventually flatten — the first bit of reasoning helps most, and each additional increment helps less. There’s a point past which more compute isn’t worth the cost.

So the relationship between thinking and quality is problem-dependent and non-monotonic: substantial gains on hard problems up to a point, little gain (or loss) on easy ones. The skill is spending thinking where it pays and withholding it where it doesn’t — which is exactly what reasoning-effort controls enable.

Reasoning effort: the dial

Because the right amount of thinking varies, reasoning models typically expose a reasoning-effort control — a way to tell the model how hard to think (low/medium/high, or a token budget for reasoning):

• It’s the accuracy/cost/latency knob. Higher effort = more thinking = better accuracy on hard problems, but more cost and latency. Lower effort = faster, cheaper, but less thorough. The dial lets you position each query on the trade-off curve deliberately.
• Match effort to the problem and stakes. Use high effort for hard, high-stakes problems where accuracy is worth the cost and the latency is acceptable; use low effort (or a standard model) for easy, high-volume, or latency-sensitive queries. The dial is how you implement “spend thinking where it pays.”
• It’s a per-query decision. Because effort is set per request, you can route different queries differently — cheap fast answers for simple traffic, deep thinking for the hard cases — optimizing cost and latency across a whole workload rather than paying maximum everywhere.

The reasoning-effort dial operationalizes everything in this post: it turns the abstract trade-off (accuracy vs cost vs latency) into a concrete control you set based on the problem. Using it well — not maxing it out by default, not leaving it low on problems that need thinking — is central to using reasoning models economically.

The engineer’s mental model

Putting the economics together, a practical framing:

• Treat accuracy as purchasable. You can now buy accuracy with compute (tokens, time). Ask whether the accuracy gain on this problem is worth the cost and latency — often yes for hard/high-stakes, often no for easy/high-volume.
• Route by difficulty. Don’t use one setting for everything. Send easy queries to cheap, fast paths (standard models or low effort) and hard queries to reasoning with appropriate effort. This is often the biggest efficiency win: not thinking hard on problems that don’t need it.
• Watch the multipliers. Reasoning tokens, best-of-N, and search all multiply cost — be deliberate about how much compute a technique spends, especially at scale where a per-query multiplier becomes a large bill.
• Respect latency budgets. Some applications can’t wait for deep reasoning; the accuracy gain is irrelevant if the response is too slow to use. Fit thinking to the latency the use case allows.

Test-time compute makes thinking a purchase: reasoning tokens cost money, thinking costs latency, both scale with how much the model thinks, and more thinking helps on hard problems but is wasteful or harmful on easy ones. The reasoning-effort dial and difficulty-based routing are how you spend it wisely. Next: how to actually use reasoning models well — the prompting and usage differences that matter.

Key takeaways

• Test-time compute makes reasoning a purchase: thinking tokens are billed like answer tokens, so a reasoning model can cost several times more per query than a standard model — and sampling (best-of-N) and search multiply the token cost further; cost is now variable per query, scaling with difficulty and reasoning effort.
• Thinking is also slow — reasoning tokens are generated sequentially before the answer, so heavy reasoning means high latency, which clashes with interactive/latency-sensitive uses regardless of accuracy gains.
• More thinking is not always better: it helps substantially on hard multi-step problems (math, coding, logic, planning), barely helps on simple problems (factual lookup, classification, formatting), can hurt via overthinking on easy problems, and has diminishing returns even on hard ones.
• The reasoning-effort control (low/medium/high or a reasoning token budget) is the accuracy/cost/latency dial — set it per query to match the problem and stakes, using high effort for hard/high-stakes work and low effort (or a standard model) for easy/high-volume/latency-sensitive queries.
• The engineer’s mental model: treat accuracy as purchasable and ask if it’s worth it, route queries by difficulty (the biggest efficiency win — don’t think hard on problems that don’t need it), watch the cost multipliers at scale, and respect latency budgets.

Further reading

• Scaling LLM Test-Time Compute Optimally (Snell et al., 2024)
• Inference-time techniques (previous post)
• LLM Inference and Serving — why generation is latency-bound