Prompt caching is an LLM optimization that reuses the processed state of an identical prompt prefix, so the model does not recalculate it on later requests. It can reduce input latency and cost while still producing a new response. It differs from response caching because it saves model input work, not a previously generated answer.
Prompt Caching
What is prompt caching?
Prompt caching is an LLM optimization that reuses the processed state of the same prompt prefix, so the model does not recalculate it on later requests. It speeds repeated input processing but still creates a new answer.
How prompt caching works
- Place stable instructions, tool definitions, and reference text first.
- The provider finds a matching token sequence from an earlier request.
- It reuses the processed key-value state for that prefix.
- The model processes new user content and generates a fresh output.
Prompt caching vs. response caching vs. semantic caching
| Method | Stores | Best use |
|---|---|---|
| Prompt caching | Processed prompt prefix | Shared long context |
| Response caching | Generated answer | Identical requests |
| Semantic caching | Meaning-based match | Similar questions |
Benefits of prompt caching
It is most useful when an app repeatedly sends substantial fixed context.
- Lowers time to first token.
- Can reduce repeated-input costs where supported.
- Improves responsiveness for agents and chat tools with shared instructions.
Prompt caching limitations
A cache hit depends on how a provider implements matching and retention.
- Small prefix or ordering changes can cause a miss.
- Thresholds and expiry periods vary by provider.
- Understand provider data handling before sending secrets.
How to improve cache hit rates
Design stable context deliberately.
- Put fixed content first and user-specific data last.
- Use stable templates and tool schemas.
- Monitor cached-token and latency metrics.
Prompt caching in Emergent
When building AI features in Emergent, keep a consistent system prompt and reusable prompt template before changing user input. Strong prompt engineering and guidance on writing prompts that work help make stable instructions a practical app-design choice. Build efficient AI apps with Emergent.
Frequently Asked Questions
Your Questions, Answered
This will automatically populate, don't change
Don't change this element unless you know what you are doing
Does prompt caching return the same answer every time?
No. It reuses input processing, then the model generates a new response. Output can still vary with the user request and model settings.
What causes a prompt cache miss?
A changed prefix, altered ordering, different formatting, an expired entry, or a prompt below a provider's cache threshold can cause a miss.
When should developers use prompt caching?
Use it when many requests share long system instructions, tool definitions, documents, or other fixed context.
Start Building
on Emergent today
on Emergent today
Start Building
