KV cache is a transformer inference technique that stores prior attention keys and values so an AI model can generate new tokens without repeating earlier work. It improves response speed during text generation, but consumes memory as the prompt and conversation grow. It is distinct from a database key-value store such as Redis.
KV cache
What is a KV cache?
KV cache is a transformer inference technique that stores prior attention keys and values, letting an AI model generate new tokens without repeating earlier work. It speeds text generation but uses more memory as a prompt or conversation grows.
Here, K and V are mathematical representations inside each attention layer, not user records or application data. A KV cache is not Redis or another key-value database.
How KV caching works during LLM inference
Language models generate one token at a time. The cache preserves attention work from tokens already processed.
- The model processes the initial prompt during the prefill phase.
- Each transformer layer creates and stores key and value representations for prompt tokens.
- For the next token, the model calculates a new query, key, and value.
- It uses the new query with stored keys and values instead of recalculating the prompt.
- It appends the new token's keys and values, then repeats until generation ends.
- The serving system clears, evicts, or manages the cache when it is no longer needed.
KV cache vs. a key-value database
The names sound similar, but their jobs differ. Redis is not an LLM KV cache.
| Characteristic | LLM KV cache | Key-value database |
|---|---|---|
| Purpose | Reuse attention calculations | Store application data for retrieval |
| Data stored | Model tensors for tokens | Named values, such as sessions or settings |
| Lifetime | Usually request or conversation scoped | Often persistent or time-limited |
| Sharing | Restricted by model and privacy boundaries | Available to authorized applications |
Benefits of KV caching
KV caching is a core optimization for autoregressive generation.
- It avoids repeated computation for prior tokens, improving response speed.
- It can improve serving throughput because active requests do less redundant work.
- It supports prefix reuse when requests begin with the same stable, verified instructions.
- It does not change output quality because it reuses equivalent intermediate results.
KV cache limitations and operational risks
Faster generation comes with a memory cost that teams must plan for.
- Cache memory grows with tokens, layers, and active requests.
- When GPU memory is full, systems may reject work, shorten context, evict entries, or reduce concurrent requests.
- Offloading cache from GPU memory to CPU memory or storage can increase capacity, but transfers can slow generation.
- Cross-request prefix caching requires strict tenant isolation and careful handling of sensitive prompts.
Practical guidance for AI application builders
Use caching deliberately and measure its production effect.
- Enable request-scoped caching for streaming chat and generation experiences.
- Reuse prefixes only for common instructions validated for safe sharing.
- Keep users and tenants isolated in shared inference infrastructure.
- Measure time to first token separately from ongoing generation speed, since caching affects them differently.
- Clear request-scoped state when its context is no longer needed.
KV cache in Emergent
Builders of chat, coding, and agent experiences need to understand KV cache behavior to create responsive AI interactions within memory limits. Explore the Emergent glossary and guidance on the best AI app builders when planning an AI-powered product with Emergent.
Start Building
on Emergent today
on Emergent today
Start Building
