What is Prompt Caching in LLMs?

Each call carries the tool definitions, the system instructions and the user query. Here the user query is the survey.

Anthropic bills an input token read from its prompt cache at 0.1 times the base input price. A cached token is one the API has already processed in an earlier request.

The six questions are the same in every call, and only the answers change. How do you pay the cached price for the questions?

The survey as one API call carries it, under the tool definitions and the system prompt. First each answer (orange) sits in its blank, under its own question. Then the six answers leave the form and stand in one strip under the last question.

1. Fill in the blanks

The usual way is a string replace. The server writes each answer into its blank, in the middle of the text.

The prompt reads like a form a person filled in. Do you see a problem with this approach?

Look at the work the model does on every call. A language model reads a prompt through self-attention.

Self-attention relates the different positions of one sequence to compute a representation of it. Every token of the survey gets compared with every other token.

Every part of the prompt gets a score against every other part. The numbers are only there to show the shape. The green block is what the API processed on an earlier call and reuses, and the orange row and column, named User Params, are all the model computes for a new person.

The Transformer paper gives the cost of one self-attention layer as O(n²·d). Here n is the length of the sequence. The size of each token's representation is d.

Double the length of the prompt and that layer does four times the work.

The whole prompt pays that cost on every call. The six questions haven't changed since the last person, yet the model processes them from scratch.

That's too expensive!

2. Prompt caching

With prompt caching, the API reuses what it already processed in an earlier call. It fully processes only what's new.

There's one condition. The API matches the start of each request, called the prefix, against content it recently processed.

The match is exact. A change anywhere in the prefix recomputes everything after it.

Anthropic's API builds the prefix in a fixed order: tools, then system, then messages. The tool definitions and the system instructions rarely change, so they match.

The survey doesn't. Its first answer, the email, is written right under the first question.

The prefix ends there. Five of the six questions come after that email. The model recomputes them for every person.

3. Answers at the bottom

Keep the text of the survey fixed. Mention the answers at the bottom of the prompt, after the last question.

Now the prefix runs through the tools, the system instructions and all six questions. Only the answers are new work.

Anthropic's documentation asks for the same layout. It says to place static content at the beginning of your prompt.

The cache has a price and a clock. A write to the 5-minute cache costs 1.25 times the base input price.

The lifetime is 5 minutes by default. It's refreshed each time the cached content is used.

A very short prompt isn't cached at all. The minimum is between 512 and 4,096 tokens, depending on the model.

Don't change models mid-session

A cache also belongs to one model. In the words of the Claude Code team, prompt caches are unique to models.

Send the next survey to a different model and nothing matches. That model has to process the questions from scratch.

The Claude Code team gave a bigger example in April 2026. Say you are 100k tokens into a conversation with Opus. You have a question that is fairly easy to answer.

Switching to Haiku for it costs more than letting Opus answer. Haiku doesn't have a cache for those tokens and has to rebuild it.

Claude Code does let you switch. The next request then reads the entire conversation with no cache hits.

The team's advice is a subagent. Opus prepares a hand-off message for the other model on the task it needs done.

Static at the top, answers at the bottom

Whenever your server sends a request to a large language model, make most of the prompt static. Place the static parts at the top.

A server sends one request to an LLM. First the request has three parts: tools, system prompt and user query. Then the parts that stay the same (green) are on top, and the parameters (orange) come last.

For the survey, we recommend answers at the bottom, instead of a string replace. The prompt says the same thing to the model.

The bill is different. While responses keep arriving within 5 minutes, every call after the first reads the six questions at 0.1 times the base input price.

Summary

Anthropic bills a token read from its prompt cache at 0.1 times the base input price.
One self-attention layer costs O(n²·d) for a sequence of n tokens.
With caching, the API reuses what it already processed.
The prefix has to be identical. A change anywhere in it recomputes everything after it.
Anthropic's API builds the prefix in the order tools, system, messages.
A string replace ends the prefix at the first answer. Answers at the bottom keep every question inside it.
A write to the 5-minute cache costs 1.25 times the base input price.
The cache lives 5 minutes, and each use refreshes it.
Prompt caches are unique to a model.

Sources

Lessons from building Claude Code: Prompt caching is everything, Thariq Shihipar, 30 Apr 2026.
Prompt caching, Claude API documentation.
How Claude Code uses prompt caching, Claude Code documentation.
Attention Is All You Need, Ashish Vaswani and others, 2017.