HomeNews

llama.cpp Merges FP8 KV-Cache for Long-Context Inference

Bhavyadeep
Bhavyadeep
•
Nov 11, 2026 2:09 PM
•
0
 min read
Select Emergent as your Preferred news source
llama.cpp Merges FP8 KV-Cache for Long-Context Inference

💡 TL;DR

  • llama.cpp merged PR #28392 adding OCP-FP8 KV-cache support with optional runtime and static per-tensor scaling on November 11, 2026.
  • The implementation uses graph-level scaling to reduce memory footprint for long-context local inference without modifying core attention functions.
  • End-to-end tests cover conversion, loading, context shifts and correctness using NVIDIA's Mistral-Medium-3.5-128B as a validation example.

llama.cpp merged PR #28392 on November 11, 2026, adding support for OCP-FP8 KV-caches with optional runtime and static per-tensor scaling. The update targets memory-constrained long-context inference on commodity GPUs, delivering a graph-level scaling implementation that preserves the project's existing attention-layer ownership model.

Release Date and Availability

Officially released on November 11, 2026, the feature shipped via pull request #28392 authored by contributor ORippler. Reviewers ggerganov and am17an approved the merge after minor adjustments including renaming cache scale references and enabling BF16 scales alongside FP8. The change builds on llama.cpp's v0.9.0 release earlier in November, which introduced the foundational OCP FP8 E4M3 tensor type.

How FP8 KV-Cache Reduces Memory Footprint

The implementation applies FP8 quantization to the key-value cache, a memory-intensive component that grows linearly with context length. By storing cached attention states in 8-bit floating-point format instead of 16-bit or 32-bit precision, the feature cuts memory consumption roughly in half for long-context workloads. Optional runtime and static per-tensor scales allow models to preserve numerical accuracy across different quantization regimes.

ORippler's design uses graph-level scaling rather than modifying the core ggml_flash_attn_ext function. This architectural choice maintains consistency with the project's existing NVFP4 and FP8 logic, simplifying shared code paths across flash-attention and standard attention layers. The approach dequantizes FP8 values before applying rotary positional embeddings (ROPE), ensuring correctness during context shifts.

Testing and Model Compatibility

End-to-end tests cover four critical paths:

  • Conversion pipelines that transform model weights into FP8 KV-cache formats
  • Model loading routines that validate tensor shapes and scale metadata
  • Context shift operations including ROPE-shift correctness checks
  • Dequantization accuracy for FP8-scaled KV caches under varied scales

The pull request cites NVIDIA's Mistral-Medium-3.5-128B as a reference model with non-unit-scaled FP8 KV-cache, demonstrating real-world applicability beyond synthetic benchmarks. The test suite ensures that models with custom scale factors load and execute correctly across different hardware backends.

Positioning Within Local Inference Ecosystem

This merge arrives days after Apple's Harbor Thunderbolt-5 Mac clustering announcement, marking a concentrated period of local-inference memory optimization work across the ecosystem. llama.cpp's FP8 KV-cache support enables developers to run longer-context models on consumer-grade GPUs, including NVIDIA RTX 40-series and AMD Radeon 7000-series cards that previously struggled with 128K-token contexts due to VRAM constraints.

The optional scaling mechanism offers flexibility for quantization-aware trained models and post-training quantized checkpoints alike. Developers can enable runtime scaling for dynamic workloads or bake static per-tensor scales into converted model files for faster loading times.

What This Means

FP8 KV-cache support positions llama.cpp as a memory-efficient runtime for long-context applications including document analysis, multi-turn conversations and code repository reasoning. The feature democratizes access to 100K-plus token contexts on hardware that costs under $2,000, lowering the barrier for local inference deployments in privacy-sensitive environments. Organizations running on-premise AI workloads gain a path to extended context windows without upgrading to server-class GPUs or cloud instances.

About the writer

Bhavyadeepsinh Rathod is SEO Content Manager at Emergent.sh, where he covers the tools, frameworks, and workflows driving the next era of vibe coding. With 8+ years in tech content marketing, he brings a sharp SEO lens to complex subjects, making Emergent's ecosystem of AI builder tools discoverable for the builders, creators, and teams that need them most. He specializes in making complex topics feel simple, relevant, and easy to act on.

Start Building
on Emergent today
Try Emergent