The memory chip stocks that dropped 5–6% after Google published TurboQuant on March 25, 2026 were reacting to the wrong number. The 6x memory reduction is real — but what most coverage missed is the more significant claim buried in the paper: TurboQuant's error is already approaching the Shannon information-theoretic lower bound. That means there is very little room left for further compression without hurting model quality. The compression arms race for KV cache, at least in this direction, may be nearly over. That is both the most exciting and the most under-discussed aspect of what Google just published.
This piece will not rehash the headlines. Instead it will work through what PolarQuant and QJL actually do mathematically, what the benchmarks mean and where they leave gaps, why the Jevons paradox argument holds here, and what TurboQuant's proximity to the Shannon limit actually implies for everyone building on top of transformer models.
The KV Cache Problem: Why Memory Runs Out Before Compute Does
Every transformer-based language model has the same memory accounting problem. When it generates a response, it computes a key vector and a value vector for every single token it has already processed. Those vectors get stored in what is called the KV cache — think of it as a shelf of labeled folders, one for each word in the conversation. The label (key) tells the model what the folder contains. The contents (value) contain the relational data — what that word means in context of everything around it.
The folders don't stay small. KV cache memory scales linearly with context length. According to analysis at Winbuzzer, running a 70-billion-parameter model for 512 concurrent users can consume 512 GB of cache memory — nearly four times the memory needed for the model weights themselves. This is not a theoretical edge case. It is the actual constraint that determines how many users a given server can handle simultaneously, how long a context window can be before hardware runs out, and what inference costs look like at scale.
Traditional quantization methods that reduce KV cache size have always come with a hidden cost: they require quantization constants stored alongside the compressed values. Each constant is only a few bits, but at large context windows those bits compound. The per-block overhead that traditional vector quantization carries becomes a meaningful fraction of the memory budget at the scale AI companies actually operate at.
TurboQuant's core claim is that it eliminates that overhead entirely. Not reduces — eliminates. That is what makes the 6x figure different from incremental improvements in the same category. The prior best method (KIVI) achieved roughly 2.6x compression. Going from 2.6x to 6x is not a linear improvement — it is a category jump that required a fundamentally different geometric approach.
PolarQuant Explained: Pointing Instead of Giving Directions
Standard vector quantization stores word relationships using Cartesian coordinates — think of it as step-by-step directions. Go two blocks east, three blocks north, up five floors. Each axis requires its own value, and storing all those values precisely is what generates the memory overhead. To reconstruct anything useful you need both the compressed data and the normalization constants that describe the boundaries of the quantization grid. Those boundaries shift with every block of data, so they must be stored every time.
PolarQuant changes the coordinate system. Instead of step-by-step directions, it points at the destination and states the distance. Every vector gets converted from Cartesian (x, y, z per axis) to polar form: a radius representing the magnitude, and a set of angles representing the direction. The key mathematical insight is that after a random orthogonal rotation is applied to the data vector, the angular distributions become predictable and highly concentrated — they follow a known Beta distribution. Because the distribution is known in advance, the model can compute an optimal set of quantization buckets (using the Lloyd-Max algorithm) once, offline, before any inference happens. No per-block normalization is needed. The boundaries of the quantization grid do not change, so nothing has to be stored alongside the compressed values.
This is what Google's blog post meant when it described PolarQuant as "a new angle on compression" — the pun was both literal and functional. The angle is not just a metaphor for a fresh approach. It is the actual mechanism.
To be specific about what this achieves mechanically: PolarQuant groups pairs of coordinates from the d-dimensional vector, maps each pair onto a polar coordinate system, then recursively applies polar transformations until the entire vector is distilled into a single final radius and a collection of angles. Those angles are stored at 3 bits each. The radius, which carries most of the magnitude information, is handled separately. According to the Google Research blog post, because the angular pattern is "known and highly concentrated," the model skips the expensive per-block normalization step that traditional quantizers require — mapping data onto a fixed, predictable circular grid rather than a square grid with constantly shifting boundaries.
QJL: The 1-Bit Error Corrector That Completes the Picture
PolarQuant does the heavy lifting. But any compression step introduces some residual error. The second component, Quantized Johnson-Lindenstrauss (QJL), exists to handle what remains.
QJL uses the Johnson-Lindenstrauss Transform — a well-established mathematical technique for projecting high-dimensional data into lower dimensions while preserving distances between points. In TurboQuant's implementation, QJL takes the residual error left after PolarQuant compression and reduces each remaining vector number to a single sign bit (+1 or -1). One bit. That is the entire memory cost of the error correction layer.
The QJL algorithm then uses a special estimator that balances a high-precision query with that single-bit simplified data to accurately calculate attention scores — the mechanism that determines which parts of the input the model should weight heavily versus ignore. This corrects the bias that PolarQuant's compression introduces, which is why the combined system reports zero accuracy loss on needle-in-a-haystack benchmarks even at 3-bit quantization.
It is worth noting that QJL is not new to this paper. It was published as a companion paper at AAAI 2025. PolarQuant is scheduled for AISTATS 2026. TurboQuant is the umbrella system presented at ICLR 2026, combining both into a single deployment-ready pipeline. The Google Research paper first appeared on arXiv in April 2025, nearly a year before the press coverage this week.
What the Benchmark Numbers Actually Mean (And Where They Leave Gaps)
The numbers are strong, but they come with scope limitations that most coverage glossed over entirely.
What was tested: Gemma, Mistral, and Llama-3.1-8B-Instruct — all open-source models, all roughly in the 7–8 billion parameter range. Benchmarks included LongBench (question answering, code generation, summarization), Needle in a Haystack (finding specific meanings in large documents), ZeroSCROLLS, RULER, and L-Eval. The 8x attention speedup was measured specifically against a JAX baseline on Nvidia H100 GPUs at 4-bit mode. It is not an end-to-end inference throughput figure.
What was not tested: Models above roughly 8 billion parameters. According to analysis from Winbuzzer, this leaves open the question of whether the guarantees hold at 70B or 405B scale — exactly where KV cache sizes become the most prohibitive and where production operators would most want compression savings. Community experiments have already found that TurboQuant's quantization noise becomes more visible at smaller model sizes (under 3B parameters) and that 3-bit mode, while achieving maximum compression, shows noticeable quality degradation on models smaller than 8B. The sweet spot for most use cases appears to be 4 bits, where quality is nearly indistinguishable from FP16 on models 3B and above.
The competition that nobody mentioned: TurboQuant is not the only KV cache compression paper at ICLR 2026. Nvidia's KVTC achieves 20x compression — more than three times TurboQuant's reduction — with less than one percentage point accuracy penalty, tested on models from 1.5B to 70B parameters. KVTC uses PCA-based decorrelation and entropy coding that borrows concepts from JPEG compression. Unlike TurboQuant's data-oblivious design, KVTC requires a one-time offline calibration step per model. The trade-off is real: TurboQuant requires zero calibration and works on any vector from any model immediately. KVTC achieves higher compression but requires preparation. Both are valid approaches for different deployment scenarios.
| Method | Compression | Calibration Required | Models Tested | Accuracy Loss |
|---|---|---|---|---|
| TurboQuant | 6x (3-bit) | None | Up to ~8B params | Zero (reported) |
| Nvidia KVTC | 20x | One-time offline (per model) | 1.5B–70B params | <1% penalty |
| KIVI (prior best) | 2.6x | Some calibration | Various | Marginal |
| FP16 (baseline) | 1x (no compression) | N/A | All models | Reference |
The Shannon Limit: Why TurboQuant May Be Close to the Ceiling
This is the part that got almost no coverage, and it is arguably the most significant technical claim in the paper.
Shannon's information theory establishes a theoretical lower bound for how much any compression system can reduce data while preserving the information it contains. You cannot compress below the Shannon limit without losing information — it is a mathematical ceiling, not an engineering constraint that can be engineered around. According to technical analysis at turboquant.net, TurboQuant's error is already close to this information-theoretic lower bound. The paper reports roughly 4x to 4.5x compression with little visible performance loss in their controlled evaluation, and the method approaches what the Shannon limit actually permits for this type of data.
The practical implication: the compression arms race for KV cache in this approach is likely near its end. Future improvements will probably come from orthogonal directions — smarter decisions about which tokens to cache (sparse attention approaches), better hardware that makes FP8 native, or architectural changes to transformers themselves. TurboQuant does not leave a lot of room for a "TurboQuant 2.0" that extracts another 6x from the same mechanism.
This also contextualizes Nvidia's KVTC differently. KVTC claims 20x compression — well above TurboQuant's 6x — but does so using PCA-based decorrelation and entropy coding rather than PolarQuant's geometric approach. If TurboQuant is near the Shannon limit for its method, then KVTC either is accessing a different portion of the information structure (plausible given the architectural difference) or is accepting accuracy loss that TurboQuant is not (also plausible — KVTC's published threshold is <1% penalty, not zero). Both papers will be at ICLR 2026 in April. The comparison will be worth watching closely.
The Stock Market Got It Backwards: Jevons Paradox and Nvidia
SK Hynix dropped 6%. Samsung dropped 5%. SanDisk dropped 5.7%. Western Digital dropped 4.7%. Micron dropped 3%. The market logic was direct: if AI models need 6x less KV cache memory, companies will buy fewer memory chips.
This logic is historically wrong in a very specific way. The Jevons paradox, first described in 1865 by economist William Stanley Jevons studying coal consumption, observes that improvements in resource efficiency typically increase total resource use, not decrease it. When steam engine efficiency improved, coal consumption went up — because cheaper, more efficient engines made coal-powered applications viable that had not been viable before. The new demand generated by falling costs outweighed the efficiency gains.
Morgan Stanley's note on TurboQuant made the same structural argument: the algorithm does not affect model training (which dominates HBM usage on GPUs), and it primarily allows systems to handle 4–8x longer context windows or significantly larger batch sizes on the same hardware. Companies will not respond to a 6x memory efficiency gain by running 6x fewer servers. They will respond by running longer contexts, larger batches, more concurrent users, and new applications that were previously cost-prohibitive.
The analogy that applies here: JPEG compression did not reduce the total amount of image data stored or transmitted. It increased it, dramatically, because making image sharing cheap enough created categories of use — social media, digital photography, email attachments — that did not exist at previous image sizes and bandwidth costs. If inference becomes 50% cheaper, the question is not whether companies reduce their GPU fleets. It is what applications become viable at half the current cost that are not currently viable at all.
There is also a Morgan Stanley point worth noting: TurboQuant targets KV cache specifically, which is inference working memory. It has no effect on model weights — the HBM (High Bandwidth Memory) that actually drives GPU demand for training. The chip stocks that dropped are primarily memory companies (DRAM, NAND). The case that training memory demand falls because of TurboQuant is not supported by how the algorithm actually works.
Who Actually Wins From This, in Order
Google, immediately. They own the infrastructure — both server farms and TPUs. Every efficiency improvement in inference is pure margin. Their search, Gemini, and Google Cloud inference costs all drop. They also get credit for publishing it openly, which they did not have to do. Google has now published both the "Attention Is All You Need" transformer paper (which created the entire modern AI industry) and TurboQuant. Both were released publicly when they could have been kept proprietary. That track record is worth noting regardless of the competitive dynamics.
API-dependent users and developers, within months. Every company running LLMs at scale — Anthropic, OpenAI, Mistral, anyone using inference providers — will eventually integrate TurboQuant or equivalent methods. When they do, the cost reduction passes through to API pricing. The Anthropic Mythos model, mentioned in the video as expensive to run, was specifically cited as a case where inference costs could drop substantially once a method like TurboQuant is integrated. Longer context windows become accessible without hardware upgrades. Agentic workflows — which chain many model calls together — become significantly cheaper to run.
Self-hosted LLM users, once official code ships. Google's official implementation is expected around Q2 2026. Community developers have already built working implementations in Triton, MLX, and llama.cpp — including a GitHub project (turboquant_plus by Tom Turney) that reports 4.6–6.4x compression using PolarQuant with Walsh-Hadamard rotation on Apple Silicon. A feature request is open on vLLM for native TurboQuant integration. For anyone running local models on hardware with constrained VRAM — a 24GB RTX 4090, for instance — the practical effect is that models which previously required multiple GPUs can fit on one, and context windows that previously hit hardware limits can extend further.
Nvidia, indirectly. The argument that Nvidia loses is weak. A company running 10 H100s can now do what previously required 60 H100s for KV cache. They are not going to sell back 50 H100s. They are going to run 6x larger models, 6x more concurrent users, or expand into application categories that did not exist at the previous economics. Nvidia's compute efficiency just increased — a multiplier on the value of existing hardware, which tends to accelerate deployment timelines and justify further hardware investment.
My Take
The 6x memory reduction number is real, but it is not the most interesting claim in this paper. The most interesting claim is that TurboQuant's compression error is already close to the Shannon information-theoretic lower bound for this data type. If that holds up under further scrutiny — and it will be scrutinized heavily at ICLR 2026 — it means that we are likely within one research cycle of the practical ceiling for this approach to KV cache compression. The next major gains will come from somewhere structurally different: sparse attention, better architecture choices, or hardware changes. That is useful information about where to direct research effort.
The benchmark scope is the legitimate reason for skepticism. All tested models are in the 7–8 billion parameter range. The KV cache problem is most severe at 70B and 405B scale — which is precisely where production operators need the gains most. The paper does not test there, and the community has not yet validated whether the zero-accuracy-loss claim survives at that scale. Until that data exists, the "zero accuracy loss" headline should carry an asterisk: verified at up to 8B parameters, unverified above that.
The stock market reaction was a category error. The companies that dropped are DRAM and NAND manufacturers. TurboQuant targets inference KV cache — working memory during text generation. It has no bearing on training memory demand, which is what primarily drives GPU and HBM procurement. The market was pricing in a reduction in demand that the algorithm's actual scope does not support. That kind of reaction suggests the analysts pricing the drop did not read the paper.
What Google did with publication is also worth naming directly. They sat on something that could cut their own inference costs substantially, and they published it openly — same as they did with the Transformer architecture paper in 2017. The cynical read is that open publication accelerates adoption across the ecosystem, which grows the market for Google Cloud. The less cynical read is that this is just how some research teams operate. Either way: the community that benefits from open AI research has gotten another significant gift, and most of the coverage spent more words on the Silicon Valley TV show joke than on the Shannon limit proximity. That says something about the current state of AI journalism.
Key Takeaways
- TurboQuant reduces KV cache memory by at least 6x and delivers up to 8x attention speedup on H100 GPUs — with zero accuracy loss on tested benchmarks, but only at up to ~8B parameter models.
- It works via two mechanisms: PolarQuant (converts Cartesian vectors to polar coordinates, eliminating per-block normalization overhead) and QJL (1-bit error correction for residual compression noise).
- No retraining, no fine-tuning, no calibration required — it can be applied to any existing model immediately.
- TurboQuant's error is reportedly near the Shannon information-theoretic lower bound — meaning further gains via the same approach are limited.
- Nvidia's KVTC (also at ICLR 2026) achieves 20x compression but requires per-model calibration and has been tested on a wider parameter range.
- The stock market drop in memory chip companies reflected a category error — TurboQuant targets inference working memory, not training memory, which drives GPU hardware procurement.
- Official Google code release expected Q2 2026. Community implementations already exist in llama.cpp, MLX, and Triton.
FAQ
Does TurboQuant work with any LLM, or only Google's models?
TurboQuant is data-oblivious — it requires no training data, calibration, or model-specific tuning. It works on any vector from any transformer architecture. Google tested it on Gemma (their own model) as well as Mistral and Llama from other organizations. Community implementations have extended it to Qwen and other open-source models. There is no inherent reason it cannot be applied to proprietary models like GPT-4 or Claude; the decision would be up to those companies' inference teams.
Why does the 8x speedup apply only to attention and not full inference?
In a transformer model, inference involves multiple stages — prefill (processing the input prompt), attention computation (the KV cache lookup and score calculation), and feed-forward network computation. TurboQuant's 8x speedup applies specifically to the attention logit computation step, measured against a JAX baseline on H100 GPUs at 4-bit mode. The feed-forward network, which dominates computation in dense models, is not affected. For decoder-only transformers at long context, attention becomes a larger fraction of total compute — so the practical end-to-end speedup is more meaningful at longer contexts than at short ones.
When will TurboQuant be available in production inference systems like vLLM or llama.cpp?
Google's official code release is expected around Q2 2026. Community implementations are already available: a llama.cpp integration with Apple Silicon Metal support exists (the turboquant_plus project on GitHub), and there is an open feature request on the vLLM project for native integration. A developer reportedly completed an MLX implementation in 25 minutes using GPT-5.4. The community timeline will likely outpace the official release, especially for local inference use cases.
What does "zero accuracy loss" actually mean in this context?
It means that on the benchmarks tested — LongBench, Needle in a Haystack, ZeroSCROLLS, RULER, and L-Eval — the model's performance scores with TurboQuant active were indistinguishable from baseline FP16. It does not mean literally zero mathematical degradation; at 3-bit quantization there is minor numerical noise. The QJL error correction component removes the bias from that noise. "Zero accuracy loss" is the benchmark result, not a claim about mathematical perfection. Community experiments have found that 4-bit is the practical sweet spot — quality essentially matches FP16 on 3B+ parameter models, while 3-bit shows noticeable degradation on smaller models.
How does TurboQuant relate to Google's DeepSeek moment comparison?
Cloudflare CEO Matthew Prince used the DeepSeek comparison publicly after TurboQuant's release. The structural parallel is real: both are efficiency improvements that make AI computation cheaper without new hardware. The difference is that DeepSeek was a training efficiency breakthrough (achieving competitive model quality at lower training cost), while TurboQuant is an inference efficiency breakthrough (reducing the memory and compute cost of running an already-trained model). DeepSeek affected training economics; TurboQuant affects production deployment economics. Both matter, but they address different parts of the cost structure.
Could this affect the pricing of Claude, ChatGPT, or other commercial AI products?
Potentially yes, but on a timeline that depends on each company's inference infrastructure decisions. TurboQuant requires no model changes — it is a drop-in optimization for the KV cache layer. Any company running transformer-based models can integrate it without retraining. If the efficiency gains hold at the model sizes those companies use in production (which are typically larger than the 8B models tested in the paper), the inference cost reduction would create competitive pressure to either improve margins or pass savings to users through lower API pricing or higher rate limits. The Anthropic Mythos model — cited specifically in the video as expensive to run — would be a case where inference cost reduction could meaningfully affect product decisions.
Also on Revolution in AI: The Claude Mythos leak situation — here is what the documents actually said versus what is still unverified. And if you are trying to decide between Claude Max and running your own API setup, the cost breakdown is more nuanced than the subscription price suggests — the Claude Max vs API cost comparison breaks it down.
TurboQuant will be presented at ICLR 2026 next month alongside Nvidia's KVTC. The full comparison of both methods under consistent benchmark conditions — including at model scales above 8B parameters — will be the first real stress test of the zero-accuracy-loss claim. The Shannon limit proximity argument is either the most important technical point in this entire announcement or a limitation of the method that more aggressive compression (like KVTC's 20x) somehow sidesteps. Watch that ICLR session. The paper that no headline is currently covering — because it requires reading past the press release — is going to be more interesting than the stock market reaction was.
0 Comments