AIForum.

Reasoning Tokens Explained: Why Some Models Think Before Answering

On this page

THE SHORT ANSWER

Reasoning tokens are the rough work a model writes before its final answer, and that rough work is real computation. This explainer covers why a fixed amount of work per token creates a ceiling, how test-time compute and verifiable rewards fit in, and what long chains cost you in latency, context, and money.
Quick facts

A reasoning model is a language model that writes rough work before it commits to an answer, and that rough work is not decoration — it is additional computation the model could not otherwise perform. The tokens it spends on that rough work are what people mean by reasoning tokens, also called thinking tokens.

This article explains what reasoning tokens are, why a model that predicts one token at a time benefits from writing them, how the approach is trained, and what it costs you in latency, context, and money. It is written for technically curious beginners and intermediate practitioners who want to reason about when a “thinking” model is worth using — including when it runs on your own hardware.

What counts as a reasoning token

Architecturally, a reasoning model is the same kind of transformer as any other large language model: it still predicts one token at a time. What differs is how it was trained and how much text it is allowed to write before committing to an answer.

When such a model receives a question, it first produces a stretch of intermediate text — trying an approach, working through steps, noticing a mistake, backtracking, checking a result — and only then produces the final answer. The tokens in that intermediate stretch are the reasoning tokens. They are part of the model’s output stream, but they are not the answer.

This distinction matters because it separates two things that are easy to conflate. A reasoning token is not a special kind of token, and it does not come from a separate module inside the network. It is ordinary generated text that happens to sit before the answer, and whose content the model can read back and build on.

Why a fixed amount of work per token is a ceiling

One of the source lessons frames the problem with an analogy: multiply 47 by 83 in your head, right now, with no paper. Most people cannot. Hand the same person a scrap of paper and they get it in about twenty seconds. The paper does not make them cleverer; it gives them somewhere to put a partial result while they work on the next one.

A model has the same limitation and the same fix. For each token it produces, it does a fixed amount of work — the same amount for “what is 2 plus 2” as for “prove this theorem”. That is a hard ceiling: some problems need more steps than fit into one token’s worth of computation.

But the model can read what it has already written. If it writes down a partial result, it can use that partial result when producing the next token. Each written line becomes a step it does not have to hold in its head, and the number of steps is no longer fixed — it is as long as the model wants to write.

The same point can be stated in architectural terms: generating intermediate reasoning gives a network of fixed depth more serial computation. The network is not deeper; the sequence of work is longer.

An illustrative reasoning chain

Consider the kind of question that separates one-shot answering from step-by-step work: how many times does the digit 7 appear when you write out every number from 1 to 100?

A model that answers immediately has to produce the number in one shot. A reasoning model instead writes something like: the units digit contributes 7, 17, 27, … 97 — ten numbers; the tens digit contributes 70 to 79 — ten more; 77 appears in both lists, which is correct because it contains two sevens; total 20. Every intermediate line is text the model can read back and build on.

The same pattern shows up in ordinary arithmetic. Given “a shop sells 3 pens for 45 rupees, what do 7 pens cost?”, a one-shot answer jumps straight to a number and is often wrong, while a chain works out the unit price and then multiplies.

These examples illustrate the mechanism rather than report measured results. The point is the structure of the work — partial results written down, then reused — not the specific numbers.

The ingredients: steps, test-time compute, and verifiable rewards

The source material describes three ingredients. The first is writing the steps: generating intermediate reasoning gives a fixed-depth network more serial computation. The same lesson goes on to cover test-time compute, self-consistency and verifiers, and reinforcement learning on verifiable rewards.

Test-time compute is the idea that you can spend more computation while generating an answer rather than only during training. Instead of making the model bigger, you let it write more before answering.

Self-consistency and verifiers are ways of using that extra computation more reliably: sample more than one reasoning chain, or check candidate answers against something that can actually be checked.

Reinforcement learning on verifiable rewards is how a model learns to make its reasoning useful rather than merely long. When a task has an answer that can be automatically checked — a maths result, a piece of code that runs — the training signal can reward chains that lead to correct answers. This is the part that turns “writing a lot” into “thinking well”.

The headline evidence, and what it does not say

The clearest public datapoint in the sources is a benchmark comparison. On the 2024 American Invitational Mathematics Examination, a hard competition for the best high-school mathematicians, GPT-4o solved 12% of the problems. In September 2024, OpenAI released o1, which solved 74% with a single attempt. The difference OpenAI described was not a bigger model but a different habit: before answering, o1 writes a long, private chain of reasoning, and it was trained to make that reasoning useful.

Read that carefully. It is a provider-described result on one competition benchmark, measured with a single attempt. It is strong evidence that reasoning tokens can change capability. It is not a general claim that any reasoning model beats any non-reasoning model on your task, and it says nothing about cost, latency, or behaviour outside mathematics.

What reasoning tokens are not

  • Not a guarantee of correctness. A chain can be long, confident, and wrong. Training on verifiable rewards makes correct chains more likely, not certain.
  • Not a different architecture. The underlying model is still a transformer predicting one token at a time.
  • Not always visible. The source material describes o1’s chain as private. Whether a given model or API exposes its reasoning tokens varies, so check the documentation for the model you are using rather than assuming.
  • Not free. Reasoning tokens are generated output. They take time to produce, they occupy context, and on hosted APIs they are usually billed.

What this costs you in practice

Because reasoning tokens are generated tokens, the practical consequences follow directly from the mechanism.

  • Latency. The model must generate the chain before it can finish the answer. A response that writes hundreds of intermediate tokens takes longer than one that writes ten.
  • Context. The chain occupies space in the context window. On a long task, or a task with a large prompt, a long chain can crowd out the material the model needs — or push the request past the limit.
  • Memory on local hardware. When you run a model yourself, the key/value cache grows with the length of the generated sequence. Longer chains mean more cache to hold, on top of the weights themselves. Treat weight size as a floor, not the total memory figure.
  • Cost. On paid APIs, generated tokens are typically billed. A reasoning model can be the better buy per successful answer and the worse buy per call. Check current pricing for the specific model rather than assuming a rule.

Running reasoning models locally: what to watch

If you serve a reasoning model on your own machine, the chain is not just a billing line — it is a dominant variable in how the run behaves.

  1. Set the context budget deliberately. Decide how much of the window is for your prompt, how much for the chain, and how much for the final answer.
  2. Cap generation. A maximum output length that is too low will truncate the chain before it reaches an answer; one that is too high lets a model wander.
  3. Measure wall-clock time per answer, not tokens per second alone. A model with high throughput that writes long chains can still feel slow.
  4. Compare against a non-reasoning model on your actual task. The benchmark gap above is a reason to test, not a reason to assume.

These are engineering considerations that follow from how reasoning tokens work. They are not the result of a test run reported here, and the right numbers depend on your model, quantization, runtime, and hardware.

How to test whether thinking helps your task

A small, honest experiment beats a benchmark headline. The protocol below is deliberately simple so that you can reproduce it and share the result.

  1. Pick 20–50 real prompts from your own work, with answers you can check.
  2. Fix the settings you can control: same prompts, same order, same sampling parameters, same context.
  3. Run each prompt once with the reasoning mode on (or the reasoning model) and once with it off (or a comparable non-reasoning model).
  4. Score only the final answer. Do not reward a chain for looking thorough.
  5. Where the interface allows it, run each prompt several times and use self-consistency — the answer supported by most chains — as the result.
  6. Record latency and token counts alongside accuracy, because a model that is right 5% more often but three times slower may not be the right default.

If your task has no checkable answer — open-ended writing, brainstorming, judgement calls — verifiable rewards give you less to work with, and the case for long chains is weaker. That is a limitation of the method, not a defect in your setup.

The short version

Reasoning tokens are the model’s scratch paper. They exist because a transformer does a fixed amount of work per token, and writing intermediate results lets it chain more of that work together. They are trained for, not merely prompted for, and that training depends on tasks whose answers can be checked. They improve capability on exactly those tasks, at the cost of latency, context, and — on hosted APIs — money.

The practical decision is not “reasoning models or not”. It is which of your tasks has a checkable answer and enough difficulty to justify the extra tokens. Start there.

If you have run a reasoning model locally, or compared thinking and non-thinking modes on a real workflow, share your setup, the task, and what you measured in the community. Environment details, token counts, and latency figures are what make those comparisons useful to everyone else.

FAQ

Are reasoning tokens the same as chain of thought?

In practice, yes: the reasoning tokens are the tokens that carry the chain of thought. The phrase 'reasoning tokens' emphasises that the chain is generated output with a cost; 'chain of thought' emphasises the structure of the steps.

Does a longer chain mean a more accurate answer?

No. Length is not correctness. Training on verifiable rewards makes correct chains more likely, but a model can still produce a long, confident, wrong chain.

Can I see a model's reasoning tokens?

It depends on the model and API. The source material describes o1's chain as private, while other models expose reasoning output. Check the documentation for the specific model you are using.

Do reasoning tokens cost more?

They are generated output tokens, so they add latency, consume context window, and are usually billed on paid APIs. The useful comparison is cost per successful answer, not cost per call.

References

  1. https://www.explain-ai.org/reasoning-models
  2. https://learn.pranaymahendrakar.com/learn/post-training/reasoning-tokens

Related terms

Reasoning token (thinking token)
A token generated as intermediate rough work before the final answer; it is part of the output stream but not the answer itself.
Chain of thought
The sequence of intermediate reasoning steps a model writes before committing to an answer.
Test-time compute
Computation spent while generating an answer rather than only during training, for example by letting a model write more before it answers.
Self-consistency
Sampling several reasoning chains for the same question and selecting the answer they most support.
Verifiable rewards
A training signal drawn from answers that can be checked automatically, such as a maths result or code that runs.

Leave a Reply

Your email address will not be published. Required fields are marked *