Inference Is a Memory Problem: KV Caches, Quantization, and Faster Decoding

When an AI feature becomes slow, the intuitive response is to ask for more compute.

That is sometimes the right answer. It is not the only one, and it is often not the first constraint to investigate. Generating text is a repeated process: the service must load model weights, preserve what earlier tokens mean for attention, and produce the next token before it can begin again. At scale, moving all of that state through memory can limit progress more than the arithmetic itself.

A useful inference optimization begins with the memory budget of the request, not with a vague desire for a faster model. That budget includes model weights, the attention state held for every active sequence, the representation precision, and the amount of output the product permits.

Once those costs are visible, techniques such as quantization, KV-cache management, and speculative decoding become design choices with explicit quality, capacity, and privacy consequences.

Why Generating the Next Token Is Different From Reading the Prompt

A transformer processes an incoming prompt in a prefill phase, then produces an answer in a decode phase. Prefill can do substantial matrix work in parallel across the prompt. Decode has a more serial shape: after each token, the model needs the state from the prior sequence before it can generate the next one.

To avoid recalculating attention for every previous token on every step, the runtime retains a key-value cache, usually shortened to a KV cache. Each layer stores the key and value vectors that describe earlier tokens. New tokens reuse them when deciding what parts of the sequence deserve attention.

The cache is essential for fast generation, but it is not free. Its size grows with several independent dimensions:

KV cache bytes ~= 2 * active sequences * sequence length * layers * hidden size * bytes per value

The leading $2$ represents separate key and value storage. The equation is deliberately approximate because architectures vary, but the growth pattern is the important part. Double the concurrent conversations, maximum context, layer count, or storage precision, and the cache can grow dramatically.

This explains a counterintuitive incident pattern: a model may load comfortably on an accelerator during a local test, then fail or slow down in production with many long conversations. The model weights are fixed. The cached state is multiplied by active users, their history, retrieved documents, and output length.

Quantization Buys Capacity by Spending Numerical Detail

Quantization stores or computes values with fewer bits. Moving from a 32-bit representation to 16 bits can cut the corresponding memory footprint roughly in half. More aggressive formats can reduce it further, enabling a model to fit on less memory, allowing more model replicas, or leaving more room for KV cache.

It is tempting to treat this as an automatic discount. It is not. Precision affects different parts of a model differently, runtimes have different kernel support, and a small aggregate benchmark change can conceal a serious regression in a specific language, task, or output format.

A responsible quantization evaluation asks four questions:

1. What must fit? Measure weights, KV cache, runtime overhead, and the desired concurrent sequence count together. A model that fits alone may still be unusable under target load.
2. What is the real workload? Include long documents, tool definitions, multilingual requests, structured output, and the rare cases that matter to the product.
3. What changes in behavior? Compare task outcomes, refusals, tool arguments, schema-validity rates, and error slices, not only a generic benchmark score.
4. What hardware and runtime will actually serve it? A format that sounds efficient on paper may be slow when the selected accelerator lacks an optimized path for it.

Pruning and distillation belong in the same family of model-size decisions. Pruning removes or neutralizes parts of a trained network, while distillation teaches a smaller model to imitate a stronger one. Both can be valuable, but both deserve full product evaluation. A smaller model can preserve a familiar writing style while becoming less reliable at an uncommon tool call or a domain-specific safety boundary.

For an application team, compression is not just an infrastructure project. It can change the behavior users trust.

The KV Cache Is Also an Information Boundary

An attention cache exists because the model needs prior context. That prior context may contain private conversation history, internal documents, customer data, or a system instruction with security-sensitive routing rules.

The cache therefore needs the same design discipline as any other temporary data store:

• scope it to the correct tenant, session, model version, and authorization context;
• invalidate it when a user changes the relevant conversation or document access;
• avoid treating cached prefix reuse as permission to mix two users who happen to ask similar questions;
• keep debug dumps, crash reports, and memory snapshots out of broad-access logs;
• define retention and deletion behavior for any persistent cache layer.

Prefix caching can be safe and extremely useful when it reuses an identical shared system prompt, a public document, or a single user's unchanged conversation prefix. It becomes dangerous when cache keys are ambiguous or a shared layer exposes state across isolation boundaries. A cache hit is a performance event. It must not become a data-disclosure event.

Reduce Cache Growth by Changing What Attention Must Remember

Several model and runtime techniques reduce the memory cost of attention without simply discarding all context.

Local or windowed attention limits each token to a recent portion of the sequence. It can dramatically reduce cache size for very long inputs, but it assumes distant context is not always necessary. That can be a fine trade for a rolling transcript and a poor one for legal clauses referenced at the beginning of a document.

Multi-query and grouped-query attention let multiple query heads share key and value information. They reduce the number of KV vectors the runtime needs to retain, with an architectural tradeoff between memory efficiency and attention flexibility.

Cross-layer sharing reuses attention state between layers in architectures designed for it. It can cut the cache footprint but is not a switch an application can turn on for an arbitrary existing model.

KV-cache quantization and compression store cached values more compactly. They reclaim memory while introducing their own numerical-error and compatibility questions.

At the runtime layer, a paged cache allocator can place fixed-size blocks non-contiguously instead of reserving one large contiguous region for every request. That reduces wasted space when conversations have different lengths and complete at different times. The mental model is similar to virtual memory: a request gets the blocks it needs rather than forcing the allocator to predict its final shape up front.

None of these techniques lets a product ignore context design. An application that sends an unbounded transcript, duplicate documents, and every historical tool result is still choosing a larger memory and latency budget. Summarization, retrieval, explicit conversation state, and deletion of obsolete context are architectural levers before they are prompt-writing tricks.

Generate More Than One Token Only When You Can Verify It

Autoregressive decode appears serial because each token depends on the previous one. Speculative decoding changes the workflow without asking the main model to trust an unchecked guess.

A small draft model proposes several next tokens. The larger target model evaluates that proposed run in parallel. It accepts the longest valid prefix, supplies the next token when the draft diverges, and repeats. When draft proposals are frequently accepted, the target model advances by several tokens for work that would otherwise have yielded one.

The important word is verify. A well-designed speculative method preserves the target model's sampling behavior while using the draft model as a speed heuristic. It is not permission to quietly replace the target with a cheaper model whenever the service is busy.

Its value depends on the workload:

• A draft model that predicts the target well can improve throughput and response latency for longer generations.
• Low acceptance rates add verification work with little benefit.
• Very short outputs may finish before setup overhead pays back.
• A poor tokenizer or incompatible runtime can erase the theoretical advantage.
• The actual user-visible measure remains TTFT, token pace, completion rate, and task quality, not only tokens per second in isolation.

Other approaches exploit structure even more directly. If an output is expected to reproduce a known span from the input, a system may be able to copy or reference that span rather than regenerate it token by token. Multiple learned prediction heads can also propose future tokens for later verification. These are valuable specialized optimizations, not universal cures for factuality or poor application design.

Kernel Work Is a Last-Mile Optimization, Not a Substitute for a Budget

The runtime eventually lowers a model into operations that specific hardware can execute. Fused kernels combine steps that would otherwise write and reread intermediate values. Vectorization, tiling, and hardware-specific compiler paths can make those operations far more efficient.

That work matters. It is why a model can perform differently on two accelerators with superficially similar specifications, and why an upgrade may require new kernels before it delivers expected performance.

But a faster kernel cannot make an unbounded context, a needlessly verbose output policy, or a cross-tenant cache safe. Before reaching for low-level optimization, establish the request's real limits:

• a maximum authorized context and output length;
• a quality and safety evaluation set that represents the product;
• target concurrency and cancellation behavior;
• cache ownership and eviction rules;
• the model, precision, and hardware combination that production will use.

Those constraints turn benchmark numbers into an operational plan. They also make the next optimization obvious: reduce the memory footprint if cache capacity is the limit, improve batching if the accelerator is underfilled, shorten a response policy if generation is the cost, or use a different serving architecture if users are waiting in the wrong phase.

Optimize the State the Product Actually Needs

An AI system does not need to remember every token merely because it can accept a large context window. It needs enough relevant, authorized state to complete a user job correctly.

That is the durable lesson behind model-level inference optimization. Quantization can make a useful model fit. KV-cache design can make concurrent generation viable. Verified drafts can make sequential decoding less wasteful. Each technique has a different failure mode, and none should be selected by a generic speed claim alone.

Measure the state the request retains, the quality it produces, and the isolation it requires. Then optimize the constraint that is genuinely limiting the product.