AIForum.

Prompt Caching and Cached Tokens: How to Lower API Costs

On this page

THE SHORT ANSWER

Prompt caching lets an API reuse the saved key-value state of an unchanged prompt prefix, cutting input-token costs and time-to-first-token. This guide explains what is actually cached, how to enable it on OpenAI and Anthropic APIs, how to estimate savings with explicit assumptions, and which prefix mistakes silently destroy your hit rate.
Quick facts

Prompt caching lowers API costs by letting a model reuse the previously computed state of a prompt prefix it has already processed. Instead of recalculating the same long system instructions, reference documents, or conversation history on every request, the provider reads a saved key-value (KV) state and processes only the new input. OpenAI describes three benefits: less wasted compute, cheaper input tokens discounted up to 95%, and faster responses because less input has to be processed before generation begins. Anthropic describes the same mechanism as “resuming from specific prefixes in your prompts,” which reduces processing time and cost for repetitive tasks or prompts with consistent elements.

This article is for developers and technically curious practitioners who pay per token and want to know what actually drives a cache hit, how to enable caching on the major APIs, how to estimate savings without guessing, and where the approach breaks down.

What prompt caching actually stores

When a model processes input tokens, it calculates intermediate states known as key-value (KV) states. Those states are what let the model refer back to earlier tokens while it processes new input and generates output. Prompt caching preserves that state for a reusable prefix — the unchanged tokens at the beginning of a prompt.

The important detail is that the cache stores KV tensors, not the tokens themselves, according to OpenAI’s prompt caching guide. That distinction explains most of the practical behaviour you will observe: caching is a computation shortcut, not a transcript store. When a later request has the same prefix and finds a matching cache entry, the model skips reprocessing those tokens. It still processes any new input and still generates a new response.

The consequence is that prompt caching is a prefix cache. It only pays off when the beginning of a request is effectively identical to a request the provider has already seen. Everything after the matched prefix is billed and processed normally.

Why prefix stability determines your hit rate

Because matching happens on the prefix, the ordering of your prompt is the single biggest lever you control. Stable content — system instructions, style guides, schemas, reference documents, tool definitions — belongs at the front. Variable content — the user’s question, a request ID, today’s date, a retrieved chunk that changes per query — belongs at the end.

Two failure patterns account for a large share of disappointing cache results:

  • Volatile tokens near the top. A timestamp, a UUID, or a per-user greeting placed in the system prompt changes the prefix on every call, so nothing after it can be reused.
  • Reordering stable blocks. Two prompts with the same content in a different order are two different prefixes. Serialising a dictionary of tool definitions without a fixed key order can be enough to break reuse.

A related misconception is that keeping a conversation “session” alive guarantees cache reuse. OpenAI’s documentation is explicit that reusing context within a session can preserve a shared prompt prefix, but maintaining a session does not guarantee a cache hit. Sessions and caches are separate mechanisms.

Enabling caching on OpenAI and Anthropic APIs

The two major providers differ in how much work you have to do.

OpenAI: prompt caching is enabled by default for supported models, so there is no request-level switch to flip. The work is therefore entirely about prompt structure and measurement, not configuration. OpenAI also notes that Agents API model calls use the same prompt-caching behaviour as the Responses API, so the same prefix discipline applies across those surfaces.

Anthropic offers two documented approaches, per its prompt caching documentation:

  • Automatic caching — add a single cache_control field at the top level of the request. The system applies the cache breakpoint to the last cacheable block and moves it forward as the conversation grows. This is described as best for multi-turn conversations where the growing message history should be cached automatically.
  • Explicit cache breakpoints — place cache_control directly on individual content blocks for fine-grained control over exactly what gets cached.

The documented Python shape of an automatic-caching request looks like this:

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an AI assistant tasked with analyzing literary works. "
           "Your goal is to provide insightful commentary on themes, "
           "characters, and writing style.",
    messages=[
        {"role": "user", "content": "Analyze the major themes in Pride and Prejudice."}
    ],
)

The same pattern is available over HTTP as a JSON body with a top-level "cache_control": {"type": "ephemeral"} alongside model, max_tokens, system, and messages. In both cases, the field is a request-level declaration; the provider decides which block becomes the breakpoint.

An illustrative cost model

Headline discounts are not a budget. To decide whether caching is worth restructuring a prompt, model it with your own numbers. The arithmetic below is illustrative only — the rates are placeholders, not quoted prices.

Assume a 12,000-token prefix (system instructions plus reference material) sent 500 times per day, and assume the cached-input rate is 10% of the standard input rate.

  • Without caching: 12,000 × 500 = 6,000,000 input tokens per day.
  • With caching and a perfect hit rate after the first request: the first request pays the normal input rate, and on providers that bill cache writes, the write rate; the remaining 499 requests read 12,000 cached tokens each, or 5,988,000 cached tokens.
  • At 10% of the standard rate, those 5,988,000 cached tokens cost the equivalent of about 598,800 standard input tokens, plus roughly 12,000 for the first request — a reduction of about 90% in input cost for that prefix.

The result is highly sensitive to the assumptions. Holding the 10% cached rate constant but dropping the hit rate to 25% cuts the reduction to roughly 22%. Raising the cached rate to 50% of standard cuts the saving to about 50%. This is why a measured hit rate matters more than a marketing number.

One pricing detail is easy to get wrong. OpenAI states that cache-write pricing is not an additive fee: input tokens are billed at the uncached-input, cached-input, or cache-write rate, not at one rate plus another. When you build a cost model, treat the three rates as mutually exclusive buckets.

Measuring instead of guessing

Both providers expose enough telemetry to check whether caching is working, and the check should come before any large prompt refactor.

OpenAI points users to a Prompt Caching Dashboard for monitoring cache read hit rates and a Prompt Cache Diagnostics tool for diagnosing cache misses and improving reuse. Those two instruments answer different questions: the dashboard tells you whether reuse is happening, and diagnostics tells you why it is not.

On Anthropic, the response usage block is the place to confirm behaviour; the documented CLI example pipes message output through a usage transform (--transform usage), which is a reminder that cache-related token counts are reported per response rather than inferred from latency.

Record three numbers per workload: total input tokens, cached input tokens, and cache-write tokens. The ratio of cached to total input is your effective hit rate. If it is far below what your prefix structure suggests it should be, move to diagnosis.

Diagnosing cache misses

Work from cheap checks to invasive ones. Each step narrows the cause.

  1. Diff two consecutive request bodies. Serialise the requests you actually sent and compare the prefix byte by byte. Most misses are visible immediately: a new timestamp, a rotated ID, a reordered JSON object, or a changed whitespace pattern.
  2. Check whether the prefix is long enough to be worth caching. Providers may define minimum cacheable prefix lengths and cache lifetimes; if your stable prefix is short, caching may be unavailable or economically pointless. Confirm the current thresholds in the provider’s documentation rather than assuming.
  3. Confirm the model and endpoint match. Cache entries are tied to the model that produced them. Switching model versions between calls, or routing the same workload across several models, splits your traffic into smaller reusable prefixes.
  4. Verify the cache is still alive. Cached prefixes expire. Anthropic’s automatic mode uses an ephemeral cache control type, and OpenAI’s documentation directs readers to current pricing and cache-write rates for model-specific behaviour. If your requests are spread thinly across the day, the prefix may expire between calls.
  5. Re-check after any prompt edit. Any change to the prefix — even a corrected typo in the system prompt — invalidates reuse until the new prefix is processed again.

If a miss persists after all five checks, the remaining explanation is usually that the prefix genuinely differs between calls. Logging the first 200 characters of the prefix alongside the cached-token count makes this visible within a day of traffic.

Tradeoffs, limits, and data retention

Caching is not free money. Cache writes can be billed at a different rate than ordinary input, so a workload with almost no repetition can cost more with caching enabled than without it. The break-even point is a function of how many times a prefix is reused before it expires.

There are also structural limits. Caching cannot help a prompt whose prefix is genuinely unique per request, which rules out many retrieval-augmented designs where the retrieved context is injected near the top. Moving retrieved chunks after the stable instructions — and after any cached conversation history — is usually the fix, but it changes what the model sees first, so it deserves a quality check rather than a blind refactor.

Data retention deserves separate attention. Anthropic’s prompt caching documentation links to its API and data retention page for how zero data retention (ZDR) applies to the feature, which signals that caching interacts with retention commitments rather than sitting outside them. Teams with contractual or regulatory constraints should read the retention documentation for their provider before enabling caching on sensitive prompts, and should confirm whether cached state counts as stored data under their own policies.

Prompt caching on local runtimes

The same prefix-reuse idea exists in self-hosted inference, where the KV cache lives in your own memory rather than a provider’s. Local inference servers generally keep a KV cache for a loaded model and can reuse it across requests when the prefix matches, but the implementation details — whether reuse is automatic, whether it must be enabled, how much memory it may occupy, and how it is evicted — differ between runtimes and versions. Treat this as a documented feature to verify in your runtime’s own documentation rather than a portable setting, and measure memory headroom, since a retained KV cache competes with the model weights for the same VRAM.

What to do next

Start with measurement, not refactoring. For one workload, log total input tokens, cached input tokens, and cache-write tokens for a day of real traffic. If the cached share is already high, your prompt structure is working and the remaining gains are small. If it is low, diff two consecutive request bodies and look for the volatile token in the prefix — that single check resolves most cases.

Once the prefix is stable, re-run the illustrative cost model with your provider’s actual published rates and your measured hit rate. Only then decide whether a larger prompt restructuring is justified.

If you have a workload where caching behaved unexpectedly — a hit rate that collapsed after a deployment, or a cost increase after enabling it — share the prompt structure and the before-and-after token counts in the Ai-Forum.org community so others can compare against their own setups.

FAQ

Does prompt caching change the model's answer?

Caching preserves the computation for an unchanged prefix, but the model still processes any new input and generates a new response. It is not a way to replay a previous answer, and it does not remove the need to send the new input.

Does keeping a conversation session guarantee a cache hit?

No. OpenAI's documentation states that reusing context within a session can preserve a shared prompt prefix, but maintaining a session does not guarantee a cache hit. Prefix matching and session continuity are separate mechanisms.

Is a cache write an extra fee charged on top of input tokens?

On OpenAI, cache-write pricing is not an additive fee. Input tokens are billed at one of three rates — uncached-input, cached-input, or cache-write — rather than one rate plus another. Check current pricing for your model, since rates vary.

How long does a cached prefix stay available?

Cache lifetimes are provider- and mode-specific, and cached prefixes expire. Anthropic's automatic caching uses an ephemeral cache control type, and OpenAI directs readers to current pricing documentation for model-specific behaviour. Verify the current lifetime for your model before designing around it.

References

  1. https://developers.openai.com/api/docs/guides/prompt-caching
  2. https://platform.claude.com/docs/en/build-with-claude/prompt-caching

Related terms

KV state (key-value cache)
The intermediate tensors a model computes while processing input tokens, which prompt caching preserves for a reusable prefix. The cache stores these states, not the tokens themselves.
Prompt prefix
The unchanged tokens at the beginning of a prompt. Cache matching operates on this prefix, so stable content should be placed first and variable content last.
Cache breakpoint
The position in a prompt up to which content is cached. Anthropic's automatic caching applies the breakpoint to the last cacheable block and moves it forward as a conversation grows.
Cache hit rate
The share of input tokens served from cache rather than processed fresh. OpenAI provides a Prompt Caching Dashboard for monitoring cache read hit rates.
Cache write
The act of storing a prefix's state so later requests can reuse it. On OpenAI, cache-write pricing is not an additive fee: input tokens are billed at either the uncached-input, cached-input, or cache-write rate.

Leave a Reply

Your email address will not be published. Required fields are marked *