The price per token for LLM APIs is not arbitrary; it reflects the KV‑Cache bandwidth required to serve a given context length.
Services charge a lower rate for short contexts (≤20k tokens) because the KV‑Cache can be kept in fast HBM, but once the context exceeds ~200k tokens, the cache must spill to slower memory tiers, increasing latency and cost.
Thus, the 200k‑token threshold seen in pricing tables is a direct consequence of hardware constraints.
근거
“The 200k context threshold is where the system switches from HBM to slower memory, and the price jumps accordingly.”
— 2026‑06‑05 Ep96
연결된 생각
- 20260605-llm-serving-latency-cost-model — {가격이 지연에 어떻게 연결되는가}
- 20260605-prefill-vs-decode-workload — {KV‑Cache가 decode에 미치는 영향}