When the KV‑Cache grows beyond ~200k tokens, the system shifts from being memory‑bound to compute‑bound.
Initially, the dominant cost is the time to fetch the entire model and the KV‑Cache from HBM, but as the context length increases, the per‑token compute cost (attention + MLP) dominates.
This transition explains why pricing tiers jump at 200k tokens: the hardware can no longer keep up with the memory traffic, and the service must throttle the number of concurrent users.
근거
“When the context length exceeds 200k, the system becomes memory‑bound, and the optimal batch size drops dramatically.”
— 2026‑06‑05 Ep96
연결된 생각
- 20260605-llm-serving-latency-cost-model — {왜 t_compute와 t_mem이 중요한가}
- vllm-sglang-optimization — {KV‑Cache 관리가 비용에 미치는 영향}