Prefill (the initial prompt) can contain thousands of tokens, while decode (the next token generation) only needs one token per step.
If a server processes a large prefill all at once, it can fill the GPU’s compute pipeline, but it also blocks the decode queue, reducing overall throughput.
Modern serving systems (vLLM, SGLang) use chunked prefill and paged attention to interleave prefill and decode tokens, keeping the GPU busy and maximizing the number of users served per 20 ms cycle.
근거
“We split the prefill into small chunks and interleave them with decode tokens so that the GPU stays full and we can serve thousands of users in a 20 ms window.”
— 2026‑06‑05 Ep96
연결된 생각
- 20260605-llm-serving-latency-cost-model — {배치와 지연의 관계}
- kv-cache-management — {PagedAttention이 어떻게 메모리 효율을 높이는가}