Cache Engineering for Agents
Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, Yu Wang
AKV enables agents to directly manage their KV cache via drop and repos operations, bypassing costly re-prefills.
How can LLM agents manage their own KV cache to reduce recomputation costs without sacrificing performance?
LLM agents performing long-horizon tasks suffer from ballooning Key-Value (KV) cache costs and accuracy degradation when context exceeds model limits. Existing solutions like trimming or summarization modify the trajectory, which invalidates cached states and forces expensive re-prefilling of the remaining context. Cache engineering allows agents to directly control their active cache by dropping specific entries and repositioning keys without altering the underlying text trajectory. This approach keeps retained cache valid, eliminating the need for re-prefill. AKV implements this via a stateless serving interface and an extended radix tree that tracks both token and operation history. This system achieves a 27.7% reduction in average inference cost and a 1.5× speedup in P95 request latency compared to conventional threshold-based summarization.
Paper Primer
The core mechanism hinges on three operations: append (standard update), drop (removes specific KV entries from attention), and repos (adjusts position encodings for retained keys). By treating cache management as a programmable tool, the system allows agents to interleave these operations with token generation, effectively shaping the active cache while keeping the logical trajectory intact.
Because identical token sequences can yield different cache states depending on the history of drops and repositions, AKV uses an extended radix tree to distinguish these states. It restores missing cache via segment-batch attention, which partitions queries at operation boundaries to reconstruct the required cache state in a single prefill pass.
AKV significantly reduces inference costs while maintaining or improving task accuracy.
Across four agent benchmarks and three LLMs, AKV reduced average per-task API costs by 27.7% and improved average success rates by 1.7 points compared to standard threshold-triggered summarization.
The system provides substantial gains in serving throughput and latency.
AKV achieved 1.18× higher output throughput and a 1.51× speedup in P95 request latency by reducing the time spent on full-attention computation during decoding.
Why is standard prefix matching insufficient for this approach?
Standard systems assume the token prefix fully determines the cache state. In cache engineering, the same token prefix can result in different cache states depending on the sequence of drop and repos operations, requiring the serving engine to track operation history to ensure correct cache reuse.
What is the primary trade-off this method addresses?
It resolves the dilemma between re-prefill cost and information retention. By allowing agents to drop older, less relevant cache entries without modifying the trajectory, the system avoids the costly re-prefill triggered by traditional context-editing methods.
The Cost of Long-Horizon Agents
LLM agents waste compute by repeatedly re‑prefilling historical context.
LLM agents repeatedly re‑process the same historical context, turning KV‑cache I/O into the primary compute bottleneck. Prior approaches trim or summarize the text trajectory, but any edit invalidates downstream cached states and forces a costly re‑prefill of the retained suffix. The paper therefore frames the problem as eliminating re‑prefill while still limiting cache growth.
The KV cache stores per‑token key and value vectors so the model can attend to past tokens without recomputing their representations.
Context engineering trims or summarizes the textual history before it reaches the model, shrinking the KV cache at the cost of altering the token prefix.
**Figure 1.** Full-context inference, context engineering, and cache engineering. Full-context inference reuses all historical KV. Context engineering removes irrelevant text and recomputes the retained history. Cache engineering keeps the text history but drops selected KV entries, reusing retained states without recomputation. Nested color blocks illustrate how retained KV can still carry information from dropped history.
The bottleneck is re‑prefilling historical context.
Cache Engineering Mechanisms
Cache Engineering lets agents edit the KV cache to avoid redundant recomputation.
Directly editing the KV cache sidesteps the costly re‑prefill of historic tokens. By dropping or repositioning cache entries, agents keep useful context while avoiding duplicate attention work.
Cache Engineering gives the agent explicit knobs to prune or shift KV entries, so attention only sees the parts that matter.
How is Cache Engineering different from Context Engineering?
Context Engineering rewrites the logical token sequence (trimming or summarizing text) before it reaches the model, whereas Cache Engineering leaves the token sequence untouched and instead prunes or re‑encodes the KV entries that the model will attend to.
Drop the tool‑response entry $C_2$ (index 2): $C \leftarrow [C_1, C_3]$.
Append the new tool token $x_4$: compute $C_4$ and concatenate → $C=[C_1, C_3, C_4]$.
Repos the remaining entries so their position indices become $0,1,2$; the next token will receive index 3.
Dropping $C_2$ removes a stale key‑value pair while preserving the reasoning encoded in $C_3$, and repos ensures position encodings stay compact, preventing overflow.
**Figure 5.** Cache operation trajectories and history-dependent KV states. (a) ① appends C to A and B, then drops A's KV; ② drops A's KV before appending C; ③ starts from B alone and appends C. All three retain B and C. (b) B's KV is identical in ① and ②, both conditioned on A, but differs in ③, where B has no prefix. C's KV differs in all three: ① is computed with A and B; ② and ③ cannot attend to A but use different B caches. (c) The extended radix tree shares the A–B prefix of ① and ②, then branches according to drop timing, distinguishing the cached states used to generate D.
**Figure 2.** Attention over history. Per-token attention to assistant and tool responses, averaged over 10 BCP requests.
**Figure 4.** Recursive self-improvement on cache engineering policy, evaluated on the BCP benchmark with MiniMax-M2.7 LLM.
**Figure 7.** BCP accuracy by length with GPT-OSS-120B.
The AKV Serving System
AKV provides a stateless API letting agents edit KV caches while the server manages reuse and eviction.
LLM agents repeatedly recompute the same context because the serving layer cannot drop or move KV entries. This wasted compute is the pain AKV is built to eliminate.
AKV treats the cache as a mutable resource that agents can edit via explicit operations, while the server remains oblivious to which physical slots hold those entries.
How does AKV differ from naïvely sharing a KV cache across requests?
Naïve sharing reuses any cached fragment that matches the token prefix, even if the requests applied different drop or repos operations. AKV records those operations in the request’s logical history and matches both token and operation prefixes, so only truly compatible cache is reused.
The interface requires each request to fully describe the cache state it needs, so the server can reconstruct it without relying on any hidden physical residue.
Why must a drop operation affect only the issuing request’s attention?
Because the stateless contract says a request’s output must be reproducible from its logical description alone. If a drop altered another request’s cache, that other request would no longer be able to reconstruct its intended state, breaking isolation.
AKV augments the radix‑tree prefix matcher with special drop nodes so that cache can be reused even when intermediate segments have been logically removed.
Insert drop node $\Delta$ after token B, recording that index 0 (A) is dropped.
Prefix matching for token C follows the path A‑B‑$\Delta$‑C, reusing B’s cache but not A’s.
When token D arrives, segment‑batch attention creates two segments: segment 1 (A‑B) attends to B’s cache; segment 2 (C‑D) attends only to caches after the drop.
Eviction marks A’s cache entry as evictable because all active requests have passed the drop point.
Drop nodes let the system keep the useful part of a prefix (B) while safely discarding the part that has been logically removed (A), all without breaking reuse of the remaining cache.
What would break if drop nodes were omitted from the radix‑tree?
Prefix matching would treat two requests that differ only by a drop as identical, causing the server to reuse cache that should have been omitted. The resulting attention would incorporate stale hidden states, corrupting the output.
When a request needs the same hidden states but with different positional encodings, AKV copies the cache and rotates the keys instead of recomputing the values.
AKV copies the KV entries for B and C.
It applies a key‑rotation: the key for B is reassigned position 0, C to position 1.
Both requests now read the same values but with the required positional encodings.
When neither request needs the original positions any longer, those KV entries are evicted independently.
Key rotation avoids a full forward pass while still delivering the exact positional context each request expects.
Why not simply recompute the hidden states after a repos instead of rotating keys?
Recomputing would require an extra forward pass through the entire model, incurring the same memory‑traffic cost that AKV’s design seeks to avoid. Rotation reuses the already‑computed values, saving both compute and bandwidth.
Empirical Evaluation
AKV cuts inference cost and latency while keeping accuracy high.
The central premise is that agents waste compute repeatedly re‑processing historic context; Cache Engineering lets agents explicitly prune or reposition KV entries, avoiding that redundancy.
AKV cuts inference cost by 27.7 % compared with the Standard baseline while preserving task accuracy.
Table 1 shows a 27.7 % cost reduction averaged over three models and four benchmarks, with no drop in success rate.
**Figure 3.** GPT-OSS-120B accuracy and cost on 100 BCP tasks. Labels indicate summary thresholds (tokens) or AKV's retained tool rounds.
**Figure 9.** Task performance and serving throughput with GPT-OSS-120B on SWE-bench Verified. (a) Online evaluation on the first 100 tasks: mean cumulative pass rate versus GPU hours across three runs per method, with shaded regions indicating ±1 standard deviation. (b) Cumulative pass rate versus cumulative logical input tokens over all 500 tasks. (c) Serving throughput versus concurrency.
**Table 1.** Performance comparison of Standard, Text Engineering, and AKV methods across different concurrency levels ($C=8, 10, 12, 14, 16$).
**Table 2.** Serving performance (latency, s; TTFT, s; TPOT, ms/token; throughput, tokens/s) of GPT-OSS-120B on Top-80 SWE-Verified across concurrency (C) levels.
**Table 3.** Drop-target and repos ablations with GPT-OSS-120B on each benchmark's first 100 tasks.
**Table 4.** Agentic cache reduction on a 100-task BCP subset. CAT means Cache-As-Tool.
AKV significantly reduces latency and cost while maintaining task accuracy.
System Design and Radix Trees
This section details AKV’s segment‑batch attention and reposition scheme that enable efficient cache reuse.
When an agent drops or repositions tokens, a naïve restore would recompute hidden states for every affected position, exploding inference cost.
Segment‑batch attention groups queries into contiguous segments that share the same visible KV pages, letting the model attend within each segment in a single forward pass — like a librarian fetching all books from the same shelf in one trip rather than one at a time.
Segment 1 attends A0, B1, C2 using causal attention; all three produce KV entries.
After dropping B, Segment 2 attends D2 with keys (A0, C1, D2) — C’s position label is now 1 because of the drop.
Dropping D starts Segment 3, which attends E3 with keys (A0, C1, E3); D is no longer visible.
A second repos moves E to position 2, so Segment 4 attends F3 with keys (A0, C1, E2, F3).
Each segment sees exactly the KV state that matches the current history, so the model never recomputes hidden states for tokens that have already been cached.
**Figure 10.** Restore with segment-batch attention. The request in (a) produces the four attention segments in (b). Numbers denote query/key positions; colored cells indicate allowed attention.
Reposition‑aware caching tags each token with its current position label and reuses caches via exact or retrigger matching — like moving a file to a new folder while keeping its contents unchanged; only the path label is updated.
Initial token nodes: T(A,0), T(B,1), T(C,2).
After dropping A, the prefix becomes T(B,1), T(C,2).
Exact matching fails for B because its position changes; retrigger matching projects B to T(B) and finds a reusable cache for B’s computation.
Reposition transforms B’s KV: $K_B$′ = R(0) R(1)⁻¹ $K_B$, $V_B$′ = $V_B$.
Similarly, C is transformed to position 1, yielding $K_C$′ = R(1) R(2)⁻¹ $K_C$, $V_C$′ = $V_C$.
Reposition nodes let the system reuse the same underlying hidden representation after a token’s logical position changes, avoiding a full recomputation.
**Figure 8.** repos in the KV pool and radix tree. (a) Two requests share A and B; Request 2 drops A before appending C. (b) B and C initially occupy positions 1 and 2. (c) repos creates copies at positions 0 and 1 by re-encoding K and copying V. The original B remains shared with Request 1, while the original C becomes evictable. The tree branches at B to distinguish positional versions and records a reposition node after C. Labels $p$ denote position $p$.
**Figure 11.** Encoding drop and reposition histories in the extended radix tree. (a) Three message trajectories interleave token appends, KV drops, and reposition operations. Although their retained tokens overlap, applying the operations at different points produces different KV states. (b) AKV records each operation in the radix tree, distinguishing cache versions by position and operation history. Numbers beside token nodes denote stored position encoding indices; $R_X$ denotes repos after $X$.
AKV runs on SGLang v0.5.20 with GPT‑OSS‑120B (BF16) across eight NVIDIA A800 GPUs; both prefill and decode workers enable radix caching, while baselines use LRU eviction and AKV uses drop‑aware eviction.
The table lists various system settings and their corresponding values used for a specific experimental configuration.
To evaluate AKV we replay 80 long‑horizon SWE‑bench Verified tasks with the largest context lengths, reconstructing each turn’s token budget, using temperature 0, and measuring throughput under fixed concurrency windows; AKV’s batched restore keeps latency low even when drops and repos occur.
Extended Results and Analysis
Quantitative ablations show how each AKV component affects speed, cost, and accuracy.
We now present ablation results that isolate the contribution of each AKV component to speed, cost, and task performance.
**Figure 12.** **Latency breakdown.** Decode-step time for GPT-OSS-120B at the same batch size, broken down by operation.
AKV reduces full‑attention time by 59.8 % relative to Standard.
Figure 12 latency breakdown.
AKV lowers total decode‑step time by 16.0 % relative to Standard.
Cost savings stem from fewer cache reads and the elimination of repeated prefill operations.
**Figure 13.** **Cost breakdown.** Average cost per task on BCP with GPT-OSS-120B for Standard, Text Engineering, and AKV.
Cache‑read cost drops by roughly 17 % versus Standard.
Figure 13 cost breakdown.
Cache‑miss input cost falls by about 48 % compared with Text Engineering.
Active context length grows more slowly because AKV continuously drops older tool‑response KV while retaining the most recent K rounds and reasoning KV.
**Figure 14.** Active context length by turn. Standard and AKV on BCP over 100 turns. The dotted line marks 96k tokens.
The slower growth keeps the context well below the 96 k‑token limit, preserving compute budget for later turns.
**Figure 15.** **Attention density across message history.** Attention density per token by layer and message index averaged over 10 selected BrowseComp-Plus trajectories, divided into old reasoning, old tool responses, and recent history.
This pattern empirically justifies AKV’s policy of preferentially dropping stale tool‑response KV entries.
Integrating AKV with the Pi harness improves downstream accuracy.
Adding AKV to the native Pi harness raises accuracy by 3 %.
Compatibility experiments on BCP.
AKV further improves accuracy by 4 % over Pi with compaction.
We also evaluate AKV on the smaller GPT‑OSS‑20B model.
Table 7 reports benchmark pass rates and costs for Standard, Text Engineering, and AKV.
AKV attains the highest average pass rate of 45.2 %.
Table 7 benchmark results.
AKV’s average cost remains low at 4.8 cents.
AKV drops KV entries belonging to older tool responses while keeping the most recent K rounds and all reasoning KV, thereby limiting active context growth without discarding useful reasoning.
How does this differ from the summarization‑based eviction used in the Standard pipeline?
Summarization rewrites the entire history, discarding both tool‑response and reasoning information in bulk. The rolling‑drop policy, by contrast, only removes stale tool‑response KV, preserving reasoning KV and allowing the context to expand gradually.
Experimental Configurations
Evaluation setups, context baselines, and cost accounting for the experiments.
We adapt four public evaluation harnesses: BrowseComp‑Plus for BCP, mini‑SWE‑agent for SWE‑bench Verified, the observation‑masking scaffold for GAIA‑Text, and WebExplorer’s harness for xbench‑DeepSearch‑2510.
Document retrieval uses Qwen3‑Embedding‑8B for BCP and GLM‑5.2 for xbench‑DeepSearch‑2510; both models also have access to the optional get‑document tool that returns full‑text pages.
Answer grading relies on GLM‑5.2 as an LLM judge for BCP, GAIA‑Text, and xbench‑DeepSearch‑2510, while SWE‑bench Verified uses its official test scripts without an LLM judge.
Agent step limits are set to 100 steps for BCP, 250 for SWE‑bench Verified, 200 for GAIA‑Text, and 100 for xbench‑DeepSearch‑2510.
Our context‑engineering baselines follow Pi’s summarization pipeline with a reserve of 16 384 tokens and a keep‑recent window of 20 000 tokens; summarization runs on the agent’s own model and replaces older dialogue while preserving system and developer instructions.
Instead of Pi’s native trigger (window size minus reserve), we compact when the exact prompt token count reaches the threshold defined in Section 5.1, discarding empty summaries and handling failures according to the benchmark error protocol.
Pi provides four prompt templates; we reproduce the shared system prompt and the turn‑prefix prompt, the latter being appended to the serialized history when a turn exceeds the token budget.
API‑equivalent inference costs are computed from the token prices in Table 6, charging only the agent’s generated tokens and its context‑summary calls.
**Table 6.** Token prices for API-equivalent cost rate (USD per million tokens).
Cache‑miss, cache‑hit, and output tokens are billed separately at the rates from Table 6; AKV’s cache‑repositioning operations are conservatively charged at the cache‑hit rate.
Additional experiment results, including ablations and scaling studies, are reported in the subsequent appendix section.