Exploring a Layer-Wise Design Space for KV Cache Eviction
This paper explores the idea of varying KV cache eviction methods across different layers of a Transformer model, showing that heterogeneous routing can improve performance compared to homogeneous policies. The results suggest that layer-wise method composition is a design dimension for KV cache compression that can be exploited.
Save an API key to vote.