Prompt Caching and Prefix Caching in Local LLMs: How They Reduce Memory and Latency

Prompt caching and prefix caching are the same underlying idea applied at different layers of the stack: when a request arrives with a prompt prefix the engine has already processed, the engine reuses the stored key-value (KV) state for that prefix instead of recomputing it. In local inference this typically shows up as a lower time to first token (TTFT) and less wasted GPU work on repeated system prompts, long documents, or shared chat history.

The important caveat for anyone running models on their own hardware is that this is primarily a compute optimization. Cached KV blocks still live in memory, so prefix caching does not automatically shrink your VRAM footprint — it changes how much work you repeat, and, when many requests share a prefix, how many copies of that state you keep.

What actually gets reused: KV cache blocks

When a transformer processes input tokens, it computes intermediate states known as key-value states. OpenAI’s prompt caching documentation describes these as the states that “let the model refer back to earlier tokens while processing new input and generating output tokens.” The cache stores those tensors, not the tokens themselves.

Inference engines organize that KV cache into fixed-size blocks. vLLM’s design documentation explains the mechanism directly: it hashes each KV cache block using the tokens in the block plus the tokens in the prefix before it. In the documentation’s example, the sentence “A gentle breeze stirred the leaves as children laughed in the distance” is split into three blocks, and the third block is identified by both its own tokens (“laughed in the distance”) and the prefix that precedes it.

This is why prefix caching is prefix-shaped rather than arbitrary: reuse only works from the beginning of the prompt forward. If a request diverges in the middle, everything after the divergence point must be recomputed.

How block hashing keeps reuse correct

vLLM builds each block hash from three components:

• Parent hash value — the hash of the preceding block, which chains blocks together into a prefix.
• Block tokens — the exact token tuple in this block, included to reduce hash collisions.
• Extra hashes — values that make the block unique, such as LoRA IDs, multi-modality input hashes, and cache salts used to isolate caches in multi-tenant environments.

Two details from the vLLM documentation matter in practice. First, only full blocks are cached, so a partial trailing block is not reusable. Second, the docs note that in previous versions the hash key was not guaranteed to be collision-free, and that as of v0.11 the default hashing changed. If you are running an older build, the collision characteristics of your cache may differ from current documentation.

The chained parent hash is what makes the scheme safe: a block is only reusable if the entire prefix leading to it matches, not just the block’s own tokens.

Prompt caching vs prefix caching: one idea, two layers

The terms are often used interchangeably, but they describe different ownership models.

Prompt caching is the hosted-API framing. OpenAI’s documentation states that prompt caching is enabled by default for supported models, that it reuses work when requests share the same prompt prefix, and that cached input tokens are billed at a reduced cached-input rate discounted up to 95%. It also notes that cache-write pricing is not an additive fee: input tokens are charged at the uncached-input, cached-input, or cache-write rate. You do not manage the cache; you influence hit rates by keeping prefixes stable.

Prefix caching is the self-hosted framing. vLLM, SGLang, and other serving engines implement it locally, and you own the cache size, eviction policy, and isolation rules. vLLM’s documentation describes prefix caching as “almost a free lunch” that “won’t change model outputs” — a useful property, since it means enabling it should not alter your generations.

Where the latency and memory savings come from

Latency. The saved work is prefill: the forward pass over the prompt that fills the KV cache before the first output token is produced. Skipping prefill for a reused prefix is what reduces TTFT. The longer the shared prefix relative to the new tokens, the larger the effect. A short system prompt followed by a fresh user question gains little; a 20,000-token document reused across many questions gains a lot.

Memory. This is where expectations often overshoot. A cached block is still resident KV state. If ten requests share the same 20,000-token prefix, caching lets the engine keep one copy of that prefix’s blocks instead of ten — that is a genuine reduction in aggregate memory pressure. But if only one request uses the prefix, caching adds bookkeeping and holds blocks that might otherwise have been freed. Whether your peak VRAM goes down depends on your sharing pattern and your engine’s eviction policy, not on prefix caching alone.

For a fuller memory picture, remember that total usage is weights plus KV cache plus runtime overhead. Prefix caching changes the middle term; it does not touch the first.

Turning it on in local stacks

vLLM

vLLM exposes prefix caching as a server configuration option — for example, an --enable-prefix-caching flag on the OpenAI-compatible server. Because defaults and flag names have changed across releases (the hashing behavior changed as of v0.11), confirm the current option against the documentation for your installed version rather than assuming a flag from an older guide still applies.

llama.cpp

llama.cpp has long supported saving and reloading prompt state to a file via --prompt-cache, and a --cache-reuse option for reusing chunks of an existing prompt when context is shifted. These are not identical to vLLM’s automatic block-level prefix caching: the file-based cache is explicit and typically suited to repeated single-session workloads. Run your build’s --help output to confirm which options exist in your version.

SGLang and other serving engines

SGLang is widely associated with RadixAttention, which performs automatic prefix reuse across requests without manual cache keys. If you are choosing a serving engine specifically for shared-prefix workloads, this is one of the documented differentiators to compare.

Desktop runners

Consumer-facing runners and Ollama-style tools vary in how much of this they expose. Some cache prompt state implicitly per conversation; others expose little control. If prefix reuse is important to your workflow, verify behavior with a measurement rather than assuming it is active.

A reproducible experiment: measure TTFT with and without prefix reuse

The following is a test protocol, not a set of measured results. Run it on your own hardware to get numbers that apply to your setup.

1. Pick one model, one quantization, and one fixed context length. Record them.
2. Build a long, stable prefix — for example, a 4,000-token document — and a short, fixed question appended after it.
3. Send the request twice with prefix caching disabled. Record TTFT for the second request (the first includes model load and warmup and should be discarded).
4. Restart the server, enable prefix caching, and send the same two requests. Record TTFT for the second request.
5. Repeat each configuration at least five times and report the median and spread, not a single run.
6. As a control, send a request whose prefix differs in the first few tokens. If TTFT matches the cold case, your cache is behaving as a prefix cache should.

Report the engine and version, the flag values, the prompt lengths, and the GPU. Without those, TTFT numbers are not comparable across machines.

Why cache hits miss: common failure modes

• The prefix changes early. A timestamp, session ID, or randomized system prompt near the start invalidates everything after it.
• The trailing block is partial. vLLM caches only full blocks, so short prompts may never produce a reusable block.
• Extra hashes differ. A different LoRA adapter ID, a different image hash, or a different cache salt produces a different block key by design.
• Eviction under pressure. When KV memory fills, blocks are dropped. A cache that hit at low load may miss under concurrency.
• Multi-tenant isolation. Cache salts deliberately prevent cross-tenant reuse, which is a security feature, not a bug.
• Chat templates inject variation. If your template renders dates or tool schemas differently per request, the prefix is not actually stable.

Tradeoffs, limits, and what to do next

Prefix caching is not a substitute for quantization, smaller models, or offloading decisions — it reduces repeated prefill work, not the cost of a single cold request. It also adds memory bookkeeping and, in shared deployments, carries a side-channel consideration: cache timing can reveal whether another tenant has processed a given prefix, which is exactly why engines support cache salts for isolation.

The practical next step is to measure your own workload. If your prompts share long, stable prefixes — RAG documents, fixed system instructions, long agent histories — enable prefix caching in your serving engine and verify the TTFT change with the protocol above. If your prompts are short and varied, expect little benefit and keep your attention on KV cache size limits instead.

If you run the experiment, share your engine, version, hardware, prompt lengths, and median TTFT in the community so others can compare against a reproducible baseline.