Discussion about this post

User's avatar
Nan Deng's avatar

The failure mode I keep hitting is a stale block-to-worker index. llm-d's numbers make the incentive obvious: a 10k-token prefix drops TTFT from 4.3s to 0.6s on a warm vLLM replica, but with 8 replicas random routing only lands on that GPU about 1 time in 8, and a chained block hash means one edited token invalidates every block after it. If an eviction event over ZMQ arrives after the router already scored Worker B as holding blocks A-D, you both miss the cache and pile load onto the GPU you thought was warm. I would discount a match whose last KV event is older than that worker's cache generation, not only trade queue depth against hit length.

Ready for more?