Logo
All posts

Google's vector compression algorithm - Turboquant

In this blog we will see how Google's TurboQuant shrinks this memory to about 3 bits a number, without losing accuracy.

Gaurav Sen/12 min read

Every time you talk to an AI agent, it stores part of the conversation in its history.

Chat with it for a hundred messages. Every token of them stays in the GPU's memory. It's held there until the conversation ends.

Engineers running these models face two problems:

  1. A long conversation's memory can be as large as the model itself.
  2. Compressing that memory usually costs accuracy, and extra memory of its own.

In this blog we will see how Google's TurboQuant shrinks this memory to about 3 bits a number, without losing accuracy.

1. What the KV cache stores

What does a model keep from the earlier messages? It keeps what it worked out about every token it has read.

Meaning the first time you talk to an agent, it has a clean slate. The next message will be sent along with all of the previous conversation history.

A chat with an AI agent fills one message at a time: Book a flight, Where to, Delhi on Friday, What time, After 6pm, Window or aisle. Each goes to the LLM with every message before it, and a block piles up under the LLM for each, until the LLM is slow.
A chat with an AI agent fills one message at a time: Book a flight, Where to, Delhi on Friday, What time, After 6pm, Window or aisle. Each goes to the LLM with every message before it, and a block piles up under the LLM for each, until the LLM is slow.

To answer, the model compares each new token with every token before it. That comparison is attention.

For every token, each layer works out two vectors. A new token is compared against each key. When the comparison says a token matters, the new token reads its value.

Working them out again for every new message would be slow.

So the model stores them. This store is called the KV cache, for keys and values.

The KV cache as a grid, Message 1 to Message 6 down the side. Each message adds as many rows and columns as it has tokens, from 3 by 3 to 15 by 15: the cached tokens in green, the newest message's in orange.
The KV cache as a grid, Message 1 to Message 6 down the side. Each message adds as many rows and columns as it has tokens, from 3 by 3 to 15 by 15: the cached tokens in green, the newest message's in orange.

As you can see, the longer the conversation, the larger the KV cache.

2. How big the cache gets

How much memory is that? Take Llama 3.1 8B, one of the models Google tested TurboQuant on.

It has 32 layers, and each layer keeps 8 keys and 8 values of 128 numbers for every token.[3] At 16 bits a number, that is 128 KB a token.

ConversationKV cache at 16 bits
1,000 tokens125 MB
32,000 tokens4 GB
128,000 tokens16 GB

At 128,000 tokens, the cache is 16 GB. The model's own weights, 8 billion numbers at 16 bits, are also 16 GB.

So one long conversation can take as much GPU memory as the model. Every other user has to fit in what is left.

Now Google's TurboQuant can reduce each of these vectors stored in memory by quantizing them into smaller numbers. Meaning a 32-bit number can be compressed into let's say a 4-bit or a 8-bit number.

What's left grows when each number takes fewer bits. Storing a number in fewer bits is called quantization.

A ring of the values one number can take: solid at 32 bits, then only 16 places on it at 4 bits.
A ring of the values one number can take: solid at 32 bits, then only 16 places on it at 4 bits.

At 3 bits a number, the 16 GB cache becomes 3 GB.

GPU memory of Llama 3.1 8B: model weights 16 GB and one 128K-token conversation's KV cache, 16 GB at 16 bits. At 3 bits it takes 3 GB, leaving available memory, and four more users fit in it.
GPU memory of Llama 3.1 8B: model weights 16 GB and one 128K-token conversation's KV cache, 16 GB at 16 bits. At 3 bits it takes 3 GB, leaving available memory, and four more users fit in it.

3. Why compression usually costs accuracy

Why not just round every number to 3 bits? Because rounding throws information away.

But the problem with compression until now is that there is a loss of accuracy in the vectors.

A vector's direction carries its meaning, and attention compares directions. Round the numbers carelessly, and a key points somewhere else. The model then attends to the wrong tokens.

A query and a key from one point. Rounded carelessly, the key points somewhere else, and the angle attention reads changes.
A query and a key from one point. Rounded carelessly, the key points somewhere else, and the angle attention reads changes.

The usual fix makes it worse in another way. A few numbers in a vector are much larger than the rest. So most methods pick a scale for every small block of numbers, kept at full precision.

Google says this overhead "can add 1 or 2 extra bits per number".[1] At 3 bits a number, that's a third or more on top.

4. Step one: rotate, then round

How does TurboQuant round without a scale for every block? It changes the vector first.

TurboQuant has a unique way to solve this problem. It first randomly rotates the vectors along all of its dimensions. The result of this rotation is that the coordinates of the vectors now, the values, follow a predictable distribution.

A rotation keeps a vector's length and its angles to other vectors. It only spreads the vector's size evenly across all the coordinates. No single number stands out.

After the rotation, every coordinate of every vector follows the same bell-shaped spread around zero. The paper calls it "a concentrated Beta distribution".[2]

Since the spread is known in advance, the best rounding levels for it can be worked out once. Every number is rounded to the nearest level, and no scale has to be stored for any block.

The values of one vector after a random rotation rise as bars into a bell shape around zero. Then the four stored values appear under them: -0.75, -0.25, 0.25 and 0.75, and each bar takes the colour of the one it rounds to, green near zero and orange at the ends.
The values of one vector after a random rotation rise as bars into a bell shape around zero. Then the four stored values appear under them: -0.75, -0.25, 0.25 and 0.75, and each bar takes the colour of the one it rounds to, green near zero and orange at the ends.

The drawing uses the four levels of my reel, on a scale chosen for the drawing. Two bits pick one of four levels.

5. Step two: one bit for the error

Is the rounded vector good enough for attention? Almost.

But of course, there is still some error.

Rounding leaves a small gap between each true number and its level. On its own, that gap nudges every attention score the same way. The scores come out slightly wrong on average.

So TurboQuant spends one more bit on the gap. It applies the Quantized Johnson-Lindenstrauss (QJL) transform to what's left. Then it keeps only the sign of each result, plus or minus.

That single bit is enough to cancel the bias. The paper calls the result "an unbiased inner product quantizer": on average, the attention scores come out right.

Three numbers of the KV cache, 16 bits each, become 2 bits each for the stored value, and one more bit, plus or minus, for the error left over.
Three numbers of the KV cache, 16 bits each, become 2 bits each for the stored value, and one more bit, plus or minus, for the error left over.

At 3 bits a number, that's 2 bits for the level and 1 bit for the error. There's no scale to store at all.

6. What Google measured

Did it hold up on real models? Google tested TurboQuant on long-context benchmarks with Gemma, Mistral and Llama 3.1 8B.[1]

TestResult
Needle in a haystack: find one fact in a long textperfect scores, with the KV cache at least 6 times smaller
Quality across long-context tasksno loss at 3 bits, without any training or fine-tuning
Speed of computing attention on H100 GPUsup to 8 times faster at 4 bits than with 32-bit keys
Vector search on GloVebetter recall than product quantization and RaBitQ

The paper puts the line at 3.5 bits a number for "absolute quality neutrality", with "marginal quality degradation" at 2.5 bits.[2]

The result is the KV cache is significantly reduced in size while accuracy is reasonably well maintained, leaving more GPU memory for longer conversations and parallel users.

The same trick works beyond the cache. A vector search engine stores billions of vectors. TurboQuant compresses them with "indexing time to virtually zero", because it doesn't learn anything from the data first.

Summary

  1. The KV cache stores a key and a value vector for every token of a conversation, in every layer.
  2. For Llama 3.1 8B, a 128,000-token conversation's cache is 16 GB at 16 bits, as large as the model's weights.
  3. Most quantization methods store a full-precision scale for every block of numbers, 1 to 2 extra bits a number.
  4. TurboQuant rotates each vector at random. Every coordinate then follows one known bell-shaped spread, and is rounded to levels worked out once.
  5. It spends 1 more bit on the rounding error: a sign from the QJL transform. That keeps attention scores unbiased.
  6. Google measured perfect needle-in-a-haystack scores with a cache at least 6 times smaller, and no loss at 3 bits.

Sources

  1. TurboQuant: Redefining AI efficiency with extreme compression, Amir Zandieh and Vahab Mirrokni, Google Research, 24 March 2026.
  2. TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate, Amir Zandieh and others, arXiv, 28 April 2025.
  3. The Llama 3 Herd of Models, Llama Team, Meta AI, 2024. Table 3: 32 layers, 8 key/value heads, model dimension 4,096 across 32 attention heads.
  4. Google's vector compression algorithm explained, Gaurav Sen, YouTube, 28 October 2026.
WhatsApp