Google's vector compression algorithm - Turboquant
In this blog we will see how Google's TurboQuant shrinks this memory to about 3 bits a number, without losing accuracy.
Every time you talk to an AI agent, it stores part of the conversation in its history.
Chat with it for a hundred messages. Every token of them stays in the GPU's memory. It's held there until the conversation ends.
Engineers running these models face two problems:
- A long conversation's memory can be as large as the model itself.
- Compressing that memory usually costs accuracy, and extra memory of its own.
In this blog we will see how Google's TurboQuant shrinks this memory to about 3 bits a number, without losing accuracy.
1. What the KV cache stores
What does a model keep from the earlier messages? It keeps what it worked out about every token it has read.
Meaning the first time you talk to an agent, it has a clean slate. The next message will be sent along with all of the previous conversation history.
To answer, the model compares each new token with every token before it. That comparison is attention.
For every token, each layer works out two vectors. A new token is compared against each key. When the comparison says a token matters, the new token reads its value.
Working them out again for every new message would be slow.
So the model stores them. This store is called the KV cache, for keys and values.
As you can see, the longer the conversation, the larger the KV cache.
2. How big the cache gets
How much memory is that? Take Llama 3.1 8B, one of the models Google tested TurboQuant on.
It has 32 layers, and each layer keeps 8 keys and 8 values of 128 numbers for every token.[3] At 16 bits a number, that is 128 KB a token.
| Conversation | KV cache at 16 bits |
|---|---|
| 1,000 tokens | 125 MB |
| 32,000 tokens | 4 GB |
| 128,000 tokens | 16 GB |
At 128,000 tokens, the cache is 16 GB. The model's own weights, 8 billion numbers at 16 bits, are also 16 GB.
So one long conversation can take as much GPU memory as the model. Every other user has to fit in what is left.
Now Google's TurboQuant can reduce each of these vectors stored in memory by quantizing them into smaller numbers. Meaning a 32-bit number can be compressed into let's say a 4-bit or a 8-bit number.
What's left grows when each number takes fewer bits. Storing a number in fewer bits is called quantization.
At 3 bits a number, the 16 GB cache becomes 3 GB.
3. Why compression usually costs accuracy
Why not just round every number to 3 bits? Because rounding throws information away.
But the problem with compression until now is that there is a loss of accuracy in the vectors.
A vector's direction carries its meaning, and attention compares directions. Round the numbers carelessly, and a key points somewhere else. The model then attends to the wrong tokens.
The usual fix makes it worse in another way. A few numbers in a vector are much larger than the rest. So most methods pick a scale for every small block of numbers, kept at full precision.
Google says this overhead "can add 1 or 2 extra bits per number".[1] At 3 bits a number, that's a third or more on top.
4. Step one: rotate, then round
How does TurboQuant round without a scale for every block? It changes the vector first.
TurboQuant has a unique way to solve this problem. It first randomly rotates the vectors along all of its dimensions. The result of this rotation is that the coordinates of the vectors now, the values, follow a predictable distribution.
A rotation keeps a vector's length and its angles to other vectors. It only spreads the vector's size evenly across all the coordinates. No single number stands out.
After the rotation, every coordinate of every vector follows the same bell-shaped spread around zero. The paper calls it "a concentrated Beta distribution".[2]
Since the spread is known in advance, the best rounding levels for it can be worked out once. Every number is rounded to the nearest level, and no scale has to be stored for any block.
The drawing uses the four levels of my reel, on a scale chosen for the drawing. Two bits pick one of four levels.
5. Step two: one bit for the error
Is the rounded vector good enough for attention? Almost.
But of course, there is still some error.
Rounding leaves a small gap between each true number and its level. On its own, that gap nudges every attention score the same way. The scores come out slightly wrong on average.
So TurboQuant spends one more bit on the gap. It applies the Quantized Johnson-Lindenstrauss (QJL) transform to what's left. Then it keeps only the sign of each result, plus or minus.
That single bit is enough to cancel the bias. The paper calls the result "an unbiased inner product quantizer": on average, the attention scores come out right.
At 3 bits a number, that's 2 bits for the level and 1 bit for the error. There's no scale to store at all.
6. What Google measured
Did it hold up on real models? Google tested TurboQuant on long-context benchmarks with Gemma, Mistral and Llama 3.1 8B.[1]
| Test | Result |
|---|---|
| Needle in a haystack: find one fact in a long text | perfect scores, with the KV cache at least 6 times smaller |
| Quality across long-context tasks | no loss at 3 bits, without any training or fine-tuning |
| Speed of computing attention on H100 GPUs | up to 8 times faster at 4 bits than with 32-bit keys |
| Vector search on GloVe | better recall than product quantization and RaBitQ |
The paper puts the line at 3.5 bits a number for "absolute quality neutrality", with "marginal quality degradation" at 2.5 bits.[2]
The result is the KV cache is significantly reduced in size while accuracy is reasonably well maintained, leaving more GPU memory for longer conversations and parallel users.
The same trick works beyond the cache. A vector search engine stores billions of vectors. TurboQuant compresses them with "indexing time to virtually zero", because it doesn't learn anything from the data first.
Summary
- The KV cache stores a key and a value vector for every token of a conversation, in every layer.
- For Llama 3.1 8B, a 128,000-token conversation's cache is 16 GB at 16 bits, as large as the model's weights.
- Most quantization methods store a full-precision scale for every block of numbers, 1 to 2 extra bits a number.
- TurboQuant rotates each vector at random. Every coordinate then follows one known bell-shaped spread, and is rounded to levels worked out once.
- It spends 1 more bit on the rounding error: a sign from the QJL transform. That keeps attention scores unbiased.
- Google measured perfect needle-in-a-haystack scores with a cache at least 6 times smaller, and no loss at 3 bits.
Sources
- TurboQuant: Redefining AI efficiency with extreme compression, Amir Zandieh and Vahab Mirrokni, Google Research, 24 March 2026.
- TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate, Amir Zandieh and others, arXiv, 28 April 2025.
- The Llama 3 Herd of Models, Llama Team, Meta AI, 2024. Table 3: 32 layers, 8 key/value heads, model dimension 4,096 across 32 attention heads.
- Google's vector compression algorithm explained, Gaurav Sen, YouTube, 28 October 2026.