llama.cpp Merges FP8 KV-Cache for Long-Context Inference

llama.cpp merged PR #28392 on November 11, 2026, adding support for OCP-FP8 KV-caches with optional runtime and static per-tensor scaling. The update targets memory-constrained long-context inference on commodity GPUs, delivering a graph-level scaling implementation that preserves the project's existing attention-layer ownership model.
Release Date and Availability
Officially released on November 11, 2026, the feature shipped via pull request #28392 authored by contributor ORippler. Reviewers ggerganov and am17an approved the merge after minor adjustments including renaming cache scale references and enabling BF16 scales alongside FP8. The change builds on llama.cpp's v0.9.0 release earlier in November, which introduced the foundational OCP FP8 E4M3 tensor type.
How FP8 KV-Cache Reduces Memory Footprint
The implementation applies FP8 quantization to the key-value cache, a memory-intensive component that grows linearly with context length. By storing cached attention states in 8-bit floating-point format instead of 16-bit or 32-bit precision, the feature cuts memory consumption roughly in half for long-context workloads. Optional runtime and static per-tensor scales allow models to preserve numerical accuracy across different quantization regimes.
ORippler's design uses graph-level scaling rather than modifying the core ggml_flash_attn_ext function. This architectural choice maintains consistency with the project's existing NVFP4 and FP8 logic, simplifying shared code paths across flash-attention and standard attention layers. The approach dequantizes FP8 values before applying rotary positional embeddings (ROPE), ensuring correctness during context shifts.
Testing and Model Compatibility
End-to-end tests cover four critical paths:
- Conversion pipelines that transform model weights into FP8 KV-cache formats
- Model loading routines that validate tensor shapes and scale metadata
- Context shift operations including ROPE-shift correctness checks
- Dequantization accuracy for FP8-scaled KV caches under varied scales
The pull request cites NVIDIA's Mistral-Medium-3.5-128B as a reference model with non-unit-scaled FP8 KV-cache, demonstrating real-world applicability beyond synthetic benchmarks. The test suite ensures that models with custom scale factors load and execute correctly across different hardware backends.
Positioning Within Local Inference Ecosystem
This merge arrives days after Apple's Harbor Thunderbolt-5 Mac clustering announcement, marking a concentrated period of local-inference memory optimization work across the ecosystem. llama.cpp's FP8 KV-cache support enables developers to run longer-context models on consumer-grade GPUs, including NVIDIA RTX 40-series and AMD Radeon 7000-series cards that previously struggled with 128K-token contexts due to VRAM constraints.
The optional scaling mechanism offers flexibility for quantization-aware trained models and post-training quantized checkpoints alike. Developers can enable runtime scaling for dynamic workloads or bake static per-tensor scales into converted model files for faster loading times.
What This Means
FP8 KV-cache support positions llama.cpp as a memory-efficient runtime for long-context applications including document analysis, multi-turn conversations and code repository reasoning. The feature democratizes access to 100K-plus token contexts on hardware that costs under $2,000, lowering the barrier for local inference deployments in privacy-sensitive environments. Organizations running on-premise AI workloads gain a path to extended context windows without upgrading to server-class GPUs or cloud instances.
on Emergent today





