H₂O Attention Score KV Cache Eviction
for On-Device LLM Inference H₂O 注意力分数 KV Cache 淘汰
面向端侧 LLM 推理

Key Findings核心发现

In plain language: When an AI model generates text, it remembers all previous words in a "KV cache." On phones and laptops, memory is limited — you can't remember everything. The old approach (sliding window) simply forgets everything except the most recent words. Our approach is smarter: it tracks which words the model actually pays attention to, and keeps those important words while forgetting the less useful ones. The result? The AI stays almost as smart as having unlimited memory, while using far less. 大白话说：AI 模型生成文字时，需要"记住"之前所有的词（KV cache）。但手机和笔记本内存有限，不可能全记住。老方法（滑动窗口）简单粗暴 — 只保留最近的词，远处的全忘掉。我们的方法更聪明：追踪模型真正"关注"了哪些词，保留那些重要的词，丢掉不太有用的。结果？AI 几乎和不限内存一样聪明，但内存占用大幅减少。

Benchmark Results基准测试结果

What to look for: "Perplexity" (PPL) measures how confused the model is — lower is better. The "Baseline" column is the gold standard (unlimited memory). We want H₂O's numbers to be as close to Baseline as possible. Notice how H₂O-heavy (red column) is always much closer to Baseline than Rotating (blue column). Speed should stay roughly the same — and it does (<0.3% difference). 看什么："困惑度"（PPL）衡量模型有多困惑 — 越低越好。"Baseline" 列是金标准（不限内存）。我们希望 H₂O 的数字尽可能接近 Baseline。注意 H₂O-heavy（红色列）总是比 Rotating（蓝色列）更接近 Baseline。速度应该基本不变 — 实测确实如此（差异 <0.3%）。

3 Qwen3 models × 4 strategies × 3 runs. max_kv_size=256, 10 PPL samples, seq_len=512. 3 个 Qwen3 模型 × 4 种策略 × 3 轮运行。max_kv_size=256，10 PPL 样本，seq_len=512。

Perplexity (lower is better)困惑度（越低越好）

Generation Speed (tokens/sec, 3-run avg)生成速度（tokens/秒，3次均值）

Peak Memory (GB)峰值内存 (GB)

Architecture架构

The trick: Modern AI chips use a "fused kernel" that computes attention in one shot for speed. The problem? It calculates how much each word matters (the "score"), but immediately throws it away. We modified this kernel to save the score with just one extra instruction — like taking a photo of a receipt before throwing it away. This costs essentially nothing, but gives us the information we need to decide which words to keep. 核心技巧：现代 AI 芯片用"融合内核"一步算完注意力，速度很快。但问题是，它计算了每个词的重要程度（"分数"），算完就丢了。我们修改了这个内核，只多加一条指令把分数存下来 — 就像丢收据前先拍张照。这几乎不花任何代价，却给了我们判断"该留哪些词"的关键信息。

Budget Allocation AblationBudget 分配消融实验

The question: Given a fixed memory budget, should we keep more "important old words" or more "recent words"? Answer: keeping more important words wins. The "H₂O-heavy" row below allocates half the budget to high-attention words and half to recent ones — it achieves only +0.4% quality loss, which is 6× better than the sliding window approach (+2.4%). 问题：在固定的内存预算下，应该多保留"重要的旧词"还是多保留"最近的词"？答案：多保留重要词更好。下表中 "H₂O-heavy" 把一半预算给高注意力的词、一半给最近的词 — 只有 +0.4% 的质量损失，比滑动窗口 (+2.4%) 好 6 倍。

Qwen3-4B-4bit, 20 samples, max_kv=256, sink=4. Qwen3-4B-4bit，20 样本，max_kv=256，sink=4。

Scaling Experiments缩放实验

The big picture: We tested two important questions: (1) How does quality change as we give the model more or less memory? (2) How does quality hold up as conversations get longer? The answer: H₂O always beats the sliding window, and the advantage is biggest when memory is tight or conversations are long — exactly the scenarios that matter most on phones. 全局视角：我们测试了两个关键问题：(1) 给模型更多或更少的内存，质量如何变化？(2) 对话越来越长时，质量还能保持吗？答案：H₂O 始终优于滑动窗口，而且在内存紧张或对话很长时优势最大 — 正是手机上最重要的场景。

KV Size Sweep (Qwen3-8B-4bit, seq=2048)KV Size 扫描 (Qwen3-8B-4bit, seq=2048)

Sequence Length Sweep (Qwen3-8B-4bit, max_kv=256)序列长度扫描 (Qwen3-8B-4bit, max_kv=256)

H₂O + KV QuantizationH₂O + KV 量化

Double the savings for free: Eviction shrinks the number of cached words. Quantization shrinks the size of each word's representation. Together, they multiply: 8× fewer entries × 2× smaller per entry = 16× total compression with zero additional quality loss (8-bit). This is like decluttering your closet AND vacuum-sealing what's left. 免费翻倍：淘汰减少了缓存的词数，量化缩小了每个词的存储大小。两者相乘：条目数减少 8 倍 × 每条缩小 2 倍 = 总共 16 倍压缩，且零额外质量损失（8-bit）。就像先整理衣柜，再把留下的真空压缩。

Quantization Across Cache Sizes不同缓存尺寸下的量化表现

Qwen3-8B-4bit, seq=2048, 5 samples. Baseline PPL = 3.555. Qwen3-8B-4bit，seq=2048，5 样本。Baseline PPL = 3.555。

Fused Quantized SDPA Kernel融合量化 SDPA 内核

The bottleneck wasn't attention — it was dequantization. Quantized KV caches previously required dequantizing the entire cache to FP16 every step before the attention kernel could read it. Our new Metal kernel (sdpa_vector_quantized) reads packed integer data directly and dequantizes per-thread in registers — zero intermediate buffers, zero extra kernel launches. This recovers 8.5 percentage points of throughput. 瓶颈不在注意力计算 — 而在反量化。量化 KV 缓存此前每步都需要将整个缓存反量化为 FP16，注意力内核才能读取。我们的新 Metal 内核（sdpa_vector_quantized）直接读取打包的整数数据，在寄存器中逐线程反量化 — 零中间缓冲区，零额外内核启动。吞吐量恢复 8.5 个百分点。

Qwen3-8B-4bit, max_kv=256, 200 tokens, 3-run average. All strategies measured via unified stream_generate with prompt_cache. Qwen3-8B-4bit，max_kv=256，200 tokens，3 次平均。所有策略均通过统一的 stream_generate + prompt_cache 测量。

On-Device Deployment端侧部署

Does it actually work on a phone? Yes. We ran Qwen3-4B-4bit on a real iPhone 15 Pro Max and iPad Air M3. H₂O (dense) adds zero speed overhead on both devices (19.7 TPS on iPhone, 37.9 on iPad). H₂O+8bit (without fused kernel) incurs −8% on iPhone and −23% on iPad at max_kv=256 due to per-step dequantization. The fused quantized SDPA kernel (validated on desktop at −2.9%) is expected to substantially reduce these overheads once ported to Swift. 手机上能跑吗？能。我们在真实的 iPhone 15 Pro Max 和 iPad Air M3 上跑了 Qwen3-4B-4bit。H₂O（dense）在两台设备上均零速度开销（iPhone 19.7 TPS，iPad 37.9 TPS）。H₂O+8bit（未使用融合内核）在 max_kv=256 时因每步反量化开销，iPhone 降 −8%、iPad 降 −23%。融合量化 SDPA 内核（桌面端验证仅 −2.9%）移植到 Swift 后预计将大幅减少这些开销。

iPhone 15 Pro Max (A17 Pro, 8GB)

iPad Air M3 (8GB)

When to Use H₂O什么时候该用 H₂O

The honest answer: H₂O's value is NOT about speed — it's about quality under memory pressure. On a phone with 8GB RAM, the model weights alone take ~2.2GB. As conversations grow longer, the KV cache eats into remaining memory. Without any cache limit, you'll eventually crash. With a sliding window (Rotating), you survive but the AI "forgets" important early context. With H₂O, you survive AND the AI remembers what matters. 实话说：H₂O 的价值不在速度，在于内存受限时的质量。8GB 手机上，模型权重就占了 ~2.2GB。对话越长，KV 缓存越大。不限制缓存最终会崩溃；用滑动窗口（Rotating）不会崩但 AI 会"忘掉"重要的早期上下文；用 H₂O 不会崩，而且 AI 还记得关键信息。

Per-Layer Adaptive Budget Allocation逐层自适应 Budget 分配

We profiled per-layer attention patterns (entropy, concentration, variance) across all 36 layers and allocated heavy/recent budgets proportionally. Result: no significant improvement over uniform allocation — validating that uniform h=max/2 is near-optimal for Qwen3. 我们分析了全部 36 层的 attention 模式（熵、集中度、方差），按比例分配 heavy/recent budget。结果：相比均匀分配无显著改善 — 验证了 h=max/2 的均匀分配对 Qwen3 已近最优。

Future Work未来工作