The rising measurement of Giant Language Fashions (LLMs) makes environment friendly inference difficult, primarily as a result of reminiscence calls for of the autoregressive Key-Worth (KV) cache. Present eviction or compression strategies scale back value however depend on heuristics, resembling recency or previous consideration scores, which serve solely as oblique proxies for a token’s future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement studying (RL) downside: studying to rank tokens by their predicted usefulness for future decoding. To this finish, we introduce KV Coverage (KVP), a framework of light-weight per-head RL brokers educated on pre-computed technology traces utilizing solely key and worth vectors. Every agent learns a specialised eviction coverage guided by future utility, which evaluates the standard of the rating throughout all cache budgets, requiring no modifications to the underlying LLM or further inference. Evaluated throughout two totally different mannequin households on the long-context benchmark RULER and the multi-turn dialogue benchmark OASST2-4k, KVP considerably outperforms baselines. Moreover, zero-shot checks on commonplace downstream duties (e.g., LongBench, BOOLQ, ARC) point out that KVP generalizes effectively past its coaching distribution and to longer context lengths. These outcomes reveal that studying to foretell future token utility is a robust and scalable paradigm for adaptive KV cache administration.
