KIV: Run 1M token context on RTX 4070 (12GB VRAM) with zero retraining—just swap in this HuggingFace cache replacement

KIV (K-Indexed V Materialization) is a clever middleware that replaces HuggingFace's standard KV cache with a tiered retrieval system. Here's the magic: it keeps recent tokens in VRAM for speed, dumps old K/V pairs to system RAM to save space, and uses K vectors as a search index to intelligently pull back only the ~256 most relevant V entries per decode step. Tested on RTX 4070 12GB running Gemma 4 E2B (4-bit quantized): you get 1M token context with just 12MB VRAM overhead and ~6x speedup. No model retraining needed—it's a drop-in replacement that works with any model using DynamicCache. This is the kind of practical hack that makes massive context windows actually usable on consumer hardware.