Discussion about this post

User's avatar
Nan Deng's avatar

The part I keep coming back to is that disk only wins after MLA cuts the KV cache by 93.3%, and then 3FS has to stripe the prefix across many SSDs so the read can stream to the GPU layer by layer. I would still worry about the exact-prefix rule: one token inserted at the front of an agent system prompt turns a hit into a full prefill, and the 500ms versus 13s gap on a 128K prompt is exactly that miss. The 3-day retention for agent traffic versus 7 days for chat is the other choice I would want a number for, because tool traces churn faster than chat and a short TTL throws away the 56% of input tokens that were served from disk.

Ready for more?