Continuous batching changes the active request batch between generation iterations instead of holding one fixed group until the longest request finishes. In a busy LLM service, that means a request that completes can release its slot and a waiting request can begin on the next scheduling opportunity. The result can be better throughput when traffic is concurrent and output lengths vary, but it is not a free latency win: prompt processing, KV-cache capacity and scheduler limits still determine how well the system behaves.
The important distinction is that continuous batching is a scheduling policy. It is not the same thing as PagedAttention, KV-cache quantization or chunked prefill. Those techniques address related serving bottlenecks, and production inference stacks often combine them.
Continuous batching in one paragraph
A conventional fixed batch groups several requests and processes that group together. If one request finishes early while another keeps generating, the shorter request may stop contributing useful work even though the batch remains occupied. In a simplified static-batching model, newly queued requests wait until the fixed batch completes.
Continuous batching, also called iteration batching or in-flight batching in some systems, changes that rule. After a generation iteration, the scheduler can remove completed sequences and admit new ones. FriendliAI’s explanation of iteration batching, rooted in the Orca research lineage, describes the key idea as dynamically changing the requests that make up the batch while it is in progress. The concept is associated with iteration-level scheduling rather than a single product name.
Static batching versus iteration-level scheduling
Consider four requests with different output lengths. In a fixed batch, all four start together. The first request may finish after only a few decode steps, while the fourth may continue for many more. If batch membership cannot change, the finished request does not create an immediately reusable slot for a waiting request.
Iteration-level scheduling makes a different decision boundary. The scheduler revisits the running set between model iterations. A completed sequence leaves, its serving state can be reclaimed, and a queued request may be admitted if compute and memory limits allow it. That is why continuous batching is especially relevant to online serving, where requests arrive continuously rather than all at once.
The basic benefit is not mysterious: the server has more opportunities to keep useful work in the active batch. But the actual gain depends on queue depth, request-length variation, model size, hardware, memory pressure and scheduler policy. A lightly loaded server may have no waiting work to fill a newly freed slot.
A step-by-step request timeline
A simple timeline makes the mechanism clearer:
| Iteration | Active requests | What changes |
|---|---|---|
| 1 | A, B, C | All three requests make progress. |
| 2 | A, B, C | B emits its final token. |
| 3 | A, C, D | B leaves; waiting request D enters if resources permit. |
| 4 | A, C, D | The new batch continues without waiting for A or C to finish first. |
That example is intentionally simplified. Real schedulers also enforce token budgets, memory limits, fairness rules and preemption policies. The important point is that batch membership is no longer fixed for the lifetime of the longest request.
Prefill, decode and why latency changes
LLM serving is commonly discussed as two phases. During prefill, the system processes the prompt and establishes the attention state needed for generation. During decode, the model generates subsequent tokens autoregressively while reusing the stored KV cache.
These phases create different scheduling pressures. A long prompt can require substantial prefill work, while an already-running generation needs repeated decode steps to keep its token stream moving. If a scheduler admits large prefill workloads too aggressively, ongoing generations can experience worse inter-token latency even while overall throughput rises.
That is why throughput should not be evaluated alone. Time to first token (TTFT) captures how long a new request waits for its first generated token. Inter-token latency or time per output token (TPOT) describes the cadence of later tokens. A production scheduler is balancing useful batch size against these latency goals.
Continuous batching is not PagedAttention
Continuous batching decides which requests run together at a given iteration. PagedAttention addresses a different problem: how KV-cache memory is allocated and managed. AiTechJournal’s PagedAttention guide explains the memory-management side in more detail.
The two techniques complement each other. Dynamic request admission is easier to sustain when memory can be allocated and reclaimed efficiently, but a scheduler does not become continuous merely because KV memory is paged. Conversely, continuous scheduling can exist without a particular paging implementation.
This distinction matters when diagnosing performance. If throughput collapses because the GPU has no room for more active sequence state, changing admission rules alone may not solve the problem. If memory is plentiful but completed requests leave idle capacity because the scheduler cannot admit replacements promptly, memory optimization alone will not solve that problem either.
Where chunked prefill fits
Chunked prefill addresses another related issue. Instead of processing a very long prompt as one large prefill unit, the system can split that work across smaller scheduling chunks. Decode requests can then continue to make progress between portions of the long prefill workload.
This can be useful when a server handles both long prompts and latency-sensitive streaming generation. It does not mean every installation should use the same chunk size or token budget. The appropriate settings depend on model behavior, hardware, prompt distribution and latency objectives.
It is therefore useful to think of three separate controls:
| Technique | Primary problem | What changes | Main tradeoff |
|---|---|---|---|
| Continuous batching | Idle capacity from fixed batch membership | Requests can enter or leave between iterations | Scheduler complexity and latency tradeoffs under load |
| PagedAttention | KV-cache allocation and fragmentation | KV state is managed in smaller blocks rather than one oversized contiguous reservation | Memory-management overhead and implementation complexity |
| Chunked prefill | Long prompts disrupting ongoing decode work | Prefill is split across smaller scheduling units | More scheduling decisions and workload-dependent tuning |
When continuous batching helps most
The clearest use case is sustained online concurrency with uneven request lengths. If the server usually has waiting work, releasing finished sequences quickly creates an opportunity to keep useful requests in flight. The effect can be especially meaningful when one user asks for a short answer while another asks for a much longer generation.
Framework documentation also reflects this throughput-oriented goal. Xinference’s continuous-batching documentation says that, in its supported configuration, chat requests are managed by continuous batching and average throughput to a single model can increase. The same documentation warns that GPU inference can consume more memory as concurrency grows.
That combination captures the operational reality: a larger useful batch can improve device utilization, but each active sequence also carries state. Throughput gains therefore depend on having enough memory headroom to keep the additional work resident.
When to tune cautiously
Continuous batching may provide little benefit when traffic is sparse. If there is no waiting request when a slot opens, dynamic admission has nothing to add. Likewise, a service optimized for the lowest possible tail latency may prefer tighter concurrency limits than a throughput-focused batch endpoint.
Long prompts can also complicate the picture. Aggressive admission may help queue throughput while worsening token cadence for requests already decoding. Memory pressure can trigger preemption, swapping or recomputation depending on the serving stack, so a higher concurrency limit can sometimes reduce effective performance instead of improving it.
For long-context workloads, memory-oriented techniques deserve separate attention. AiTechJournal’s guide to KV-cache quantization for long-context inference covers one way teams may reduce KV memory usage; that is a different optimization from changing scheduler membership.
Framework-specific knobs without universal settings
Production frameworks expose different controls, and the names should not be treated as interchangeable. Xinference documents max_num_seqs as a way to adjust concurrency for its continuous-batching path. A larger value can admit more simultaneous work, but the documentation explicitly warns about GPU-memory use.
vLLM also illustrates why API shape matters. In a vLLM issue discussion, a project collaborator explains that the synchronous LLM class can use continuous batching internally when given a large prompt batch, while AsyncLLMEngine is the suggested path for one-at-a-time asynchronous request submission. That distinction is useful because “batching” can describe both a user supplying many prompts at once and the server independently scheduling many live requests.
Do not copy a batch-size, sequence-count or token-budget setting from another deployment and assume it will transfer. The correct value depends on model memory footprint, maximum context, prompt distribution, output length, hardware and the latency service level you are trying to protect.
How to benchmark continuous batching
A fair benchmark changes one serving variable at a time. Keep the model, weights, hardware, precision, request set and prompt/output distribution constant. Then compare the scheduler configuration using the same concurrency pattern.
At minimum, record:
- Throughput: completed requests or generated tokens per unit time.
- TTFT: how long requests wait for the first generated token.
- ITL/TPOT: how smoothly later tokens arrive.
- Queueing: how long requests wait before entering useful model work.
- GPU memory: whether higher concurrency consumes KV capacity or causes preemption.
- Error and timeout rates: whether the more aggressive scheduler introduces instability.
Run the comparison across several workload shapes rather than one average case. Include short and long prompts, short and long outputs, low and high concurrency, and a mixed distribution that resembles production. Report p50 and tail latency where possible; an average can hide a scheduler that improves throughput while making a smaller set of users wait much longer.
Most importantly, avoid attributing every gain to continuous batching. If the new configuration also changes PagedAttention behavior, prefix caching, chunked prefill, quantization or model parallelism, the result is a system-level comparison rather than a clean scheduler experiment.
Practical adoption checklist
Continuous batching is a strong default concept for busy online LLM serving, but deployment should still be evidence-driven. Confirm that the workload has enough concurrency to benefit, measure TTFT and token cadence rather than throughput alone, watch KV-cache pressure, and keep framework-specific settings tied to the version actually running.
Start with a reproducible baseline, raise concurrency gradually, and keep a rollback point. If throughput improves while tail latency, memory pressure or error rates move outside the service objective, the scheduler is over-tuned for that workload. The goal is not the largest possible batch. It is the best balance of useful GPU work, predictable latency and stable memory use for the requests your system actually serves.
