TokiRecur Launches: llama.cpp Tool Recycles KV Cache

A new open-source tool called TokiRecur has launched to accelerate large language model inference by recycling KV cache across llama.cpp sessions. The project addresses a persistent performance bottleneck: every time an LLM restarts, it traditionally recomputes attention keys and values for any repeated context, wasting compute and adding latency. TokiRecur eliminates that redundancy by preserving and reusing cache state between runs.

How KV Cache Recycling Works

During transformer inference, models generate key-value pairs for each token in the attention mechanism. These pairs are stored in what practitioners call the KV cache. When context repeats across sessions, such as system prompts in chatbots or shared document prefixes in RAG pipelines, recomputing those keys and values becomes pure overhead. TokiRecur intercepts the llama.cpp workflow to save cache snapshots and restore them on subsequent launches, allowing the model to pick up exactly where it left off.

The approach is particularly valuable for multi-turn conversations, batch processing jobs with identical preambles, and agentic workflows that reinitialize models frequently. By skipping the recomputation phase, TokiRecur can cut initialization latency by 40 to 70 percent depending on the length of reused context.

Release Date and Availability

TokiRecur was officially released on November 7, 2026, as an open-source project hosted on GitHub. The tool integrates directly with llama.cpp, the widely adopted C++ inference engine for running quantized LLMs on commodity hardware. Developers can clone the repository and start using cache recycling immediately without modifying existing llama.cpp binaries.

Technical Implementation Details

TokiRecur operates as a lightweight launcher wrapper. It serializes the internal state of the KV cache to disk at the end of each inference session, then deserializes and injects that state before the next run. The serialization format is optimized for minimal I/O overhead, using memory-mapped files where supported. Key features include:

• Automatic detection of cache staleness if model weights or configuration change
• Granular control over which cache segments to preserve or discard
• Optional compression to reduce storage footprint for long contexts
• Thread-safe handling for concurrent inference jobs

The project documentation notes that cache recycling works best when context overlap exceeds 200 tokens. Below that threshold, the disk I/O cost can outweigh the computation savings, though this varies by hardware profile.

Broader Context and Similar Efforts

TokiRecur joins a growing ecosystem of tools focused on KV cache optimization. Recent innovations include lossless compression schemes that shrink cache memory usage without accuracy loss, and hardware-aware storage strategies that extend SSD lifespan under heavy cache workloads. While those projects focus on space efficiency, TokiRecur prioritizes time efficiency by treating cache as a reusable asset rather than a disposable byproduct.

The llama.cpp community has also explored sleep mode techniques that swap models in under two seconds, but those approaches still require full context recomputation. TokiRecur complements such work by making the recomputation step itself optional.

What This Means

TokiRecur's launch signals a shift toward treating inference state as a first-class artifact worth persisting across sessions. For developers running local LLMs on resource-constrained devices, or scaling inference pipelines where every millisecond counts, cache recycling offers a straightforward path to faster response times. As LLM context windows continue to grow, tools that intelligently reuse computation will become essential infrastructure. TokiRecur provides an early, accessible implementation of that principle for the llama.cpp ecosystem.