AI Engineering: 1M context windows

There is a common misconception which I've seen among software engineers.

The expectation is that if a model has context window length of 1 million, then it can fit in 1 million tokens. And so if I pass in a request of roughly 1 million tokens, then the model is going to be able to go through that stuff and give a good response.

But this is not true.

Engineers who passed long requests faced two problems:

1. The model missed facts that were in the request.
2. Every long request cost more.

In this blog we will see why a model misses facts in a long context, and how to keep the context short.

1. A request of 1 million tokens

Say your company has an assistant that answers from its documents. A user asks it one question about the expense policy.

The policy is one section of a long document. Why not pass the whole document? The window is big enough.

A context window is all the text a model can refer to while it writes a response. Anthropic's documentation lists 14 Claude models with a window of 1M tokens.

The window holds the conversation, and the output the model writes too. An input longer than it is refused, with a 400 error.

So the size tells you what fits. It doesn't tell you what the model uses.

2. The needle in the haystack

What happens to one fact in a long request? The answer to the user's question is somewhere inside the document.

So if I give an entire book of 5,000 pages and then I ask you for a specific fact, the chance of the model getting this answer correct is inversely proportional to the length of the context window.

Meaning if you pass short relevant context to the model, the probability of it getting the answer correct is high. And if you pass irrelevant long context to the model, the probability of it getting the answer correct is low.

You can test this. Place one fact inside a long text and ask the model for it. The fact is the needle, and the text is the haystack.

Then do it again with a longer text, and with the fact deeper in it. This is called a needle in a haystack test.

A public version of the test on GitHub runs it over that grid. Its README records runs on GPT-4 with a 128K window and on Claude 2.1, both from November 2023.

What did tests like it find? Anthropic's engineers write that as the tokens in the window go up, the model's recall of them goes down. They call it context rot.

Where the fact sits matters too. In a 2023 paper, models did best with the relevant text at the start or the end of the input.

They did worst with it in the middle. The paper is titled "Lost in the Middle".

3. RAG

Now practically, where does this come up? RAG, retrieval augmented generation.

A lot of engineers, they don't do any kind of chunking of their data. So they have these company documents. Any person firing a user query, first they take the query, convert it into an embedding. And then they search in their document database: which document is closest to this query?

Here the common mistake is picking up the entire document, which may have irrelevant context. And so the chance of the model missing out on important facts mentioned in the document is high.

Your assistant works the same way. It retrieves the closest document, adds it to the request, and the model generates the answer.

It finds that document among stored vectors. Each tag in the drawing below is one, and the question lands in one region. Then the whole document lands on it.

The model now reads the whole book for one fact. That's the haystack again.

4. Chunk the documents

So how does the assistant send less? It sends a piece of the document instead of all of it.

The easiest way to avoid this is chunking of documents. You take a document, break into pieces. These are pieces which are logically separate from each other.

For example, in the financial document, if you have sections which are describing different parts of the financial process, you want to take each section as one chunk and then store that embedding in the vector database.

Here is a small expense policy with three sections. Each section becomes one chunk.

document = """# Expense policy

## Travel
Flights are booked in economy for trips under six hours.

## Vendor payments
A payment above 5 lakh rupees needs the finance head's approval.

## Reimbursements
Claims are filed within 30 days of the expense.
"""

chunks = ["## " + section.strip() for section in document.split("## ")[1:]]
for chunk in chunks:
    print(chunk.splitlines()[0], "|", len(chunk.split()), "words")

## Travel | 12 words
## Vendor payments | 14 words
## Reimbursements | 11 words

Each chunk gets its own embedding. The question gets one too, and the search compares it with every chunk.

from openai import OpenAI

client = OpenAI()


def embed(texts):
    reply = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return [item.embedding for item in reply.data]


def similarity(a, b):
    return sum(x * y for x, y in zip(a, b))  # OpenAI's embeddings have length 1, so this is the cosine


question = "Who has to approve a vendor payment of 8 lakh rupees?"
asked, *stored = embed([question] + chunks)
for chunk, vector in sorted(zip(chunks, stored), key=lambda pair: -similarity(asked, pair[1])):
    print(f"{similarity(asked, vector):.2f}", chunk.splitlines()[0])

0.79 ## Vendor payments
0.30 ## Reimbursements
0.12 ## Travel

The vendor payments chunk scores highest, so it's the one that goes to the model. The other two stay out of the context window.

5. What about agents?

Does an agent have the same problem? It fills its own window as it works.

And if you're using an agent, make sure that its memory is not overloaded with old conversation history, old actions that it took. These are standard practices.

Anthropic describes two of them. In the first, a conversation near the limit is summarised, and a new window starts from the summary. This is called compaction.

The second clears old tool results out of the window. Anthropic calls it tool result clearing.

So, can a model use 1 million tokens?

Your assistant's model can take them all into its window. It doesn't read them all equally well, and each one is paid for.

Pinecone's chunking guide notes that large chunks may increase latency and cost. On Claude Haiku 5.5, prompts over 100,000 tokens cost more.

Chunking, compaction and tool result clearing all send the model less.

This is a good way to keep your context window short, which improve the accuracy and reduce the cost of your AI app.

Summary

1. A 1M-token window says what fits. An input longer than it is refused with a 400 error.
2. As the tokens in the window go up, the model's recall of them goes down. Anthropic calls this context rot.
3. A needle in a haystack test hides one fact in a long text and asks the model for it.
4. Models read the middle of a long input worst.
5. In RAG, passing the whole document is the common mistake.
6. Chunking breaks a document into pieces that make sense on their own, each with its own embedding.
7. Only the closest chunks go to the model.
8. An agent's window is kept short with compaction and tool result clearing.
9. A short context is more accurate and costs less.

Sources

1. Context windows, Claude API documentation.
2. Effective context engineering for AI agents, Anthropic Engineering, 2025.
3. gkamradt/needle-in-a-haystack: Doing simple retrieval from LLM models at various context lengths to measure accuracy, GitHub repository.
4. Lost in the Middle: How Language Models Use Long Contexts, Nelson F. Liu and six co-authors, Transactions of the Association for Computational Linguistics, 2023.
5. Chunking Strategies for LLM Applications, Roie Schwaber-Cohen and Arjun Patel, Pinecone.
6. Vector embeddings, OpenAI API documentation.