Batch Inference Throughput and Latency Calculator

LLM batch inference latency, capacity, and token-cost estimates

LLM batch inference groups several requests so that one GPU execution can serve them together. This calculator models how that batch size affects per-request latency, token throughput, request throughput, and hardware cost per 1,000 tokens when you supply a realized single-request token rate, request lengths, and fixed batch overhead.

The batch-inference model is deliberately lightweight: it uses one effective token-processing rate and one fixed overhead for each dispatched batch. It is useful for quick GPU capacity planning, comparing batch-size choices, and checking whether an LLM latency objective is plausible before testing the serving stack. Production decisions should still be validated with measured P95/P99 latency and representative traffic.

How to use the LLM batch inference calculator

1. Enter the single-request token processing rate in tokens/sec. Use a realized rate measured for one request on the hardware and runtime you expect to serve. If you only have a batch benchmark, you can approximate this rate by dividing batch token throughput by batch size.
2. Enter prompt tokens and completion tokens per LLM request. Choose typical, P50, or P95 lengths according to the capacity or latency question you are evaluating. For a tail-latency plan, use a conservative token count.
3. Choose a batch size. A larger LLM inference batch raises the modelled throughput, but can be constrained by VRAM and may increase real-world waiting time. With dynamic batching, the realized batch can be smaller when traffic is light.
4. Set per-batch overhead in milliseconds. This represents fixed work such as collation, padding, kernel launches, synchronization, and framework processing.
5. Enter hardware cost per hour in USD, using a cloud GPU price or an internal amortized cost.
6. Click Compute to calculate the batch-inference metrics. Use Copy Result to copy the displayed table into a planning document, ticket, or runbook.

For a dynamic LLM batcher that dispatches at size B or when a timer expires, compare several values such as B=1, B=4, and B=8. The comparison shows the capacity available when traffic actually fills each batch.

LLM batch inference formulas and modelling assumptions

For this LLM batching calculation, let R be the single-request token processing rate (tokens/sec), P prompt tokens per request, C completion tokens per request, B batch size (requests), O per-batch overhead (ms), and K hardware cost ($/hour).

Tokens per request: $$T_r=P+C$$

Processing time per request (seconds): $$T_{\text{proc}}=\frac{T_r}{R}$$

Total time for one LLM inference batch (seconds): $$T_{\text{batch}}=\frac{O}{1000}+T_{\text{proc}}$$ In this simplified batch model, per-request latency equals the batch time because every request in that batch completes together.

LLM token throughput (tokens/sec): $${\text{Throughput}}_{\text{tokens}}=R\times B$$

LLM request throughput (requests/sec): $${\text{Throughput}}_{\text{req}}=\frac{R}{T_r}\times B$$

Batch hardware cost and cost per 1,000 LLM tokens: $${\text{Cost}}_{\text{batch}}=K\times \frac{T_{\text{batch}}}{3600}$$ $${\text{Cost}}_{1k}=\frac{{\text{Cost}}_{\text{batch}}\times 1000}{B\times T_r}$$

This LLM batching estimate assumes near-linear token-throughput scaling with batch size, fixed overhead per batch, no explicit queueing delay, and one constant rate for both prompt processing and generation. Those assumptions can diverge from observed performance when sequence lengths, model architecture, or the serving framework change.

Worked example: an eight-request LLM inference batch

Consider a measured single-request rate of R = 40 tokens/sec, with P = 100 prompt tokens and C = 50 completion tokens per request. The LLM server dispatches B = 8 requests at once, has O = 30 ms of fixed batch overhead, and uses hardware costing K = $2.50/hour.

• Tokens per request: Tr = 100 + 50 = 150 tokens.
• Processing time: Tproc = 150 / 40 = 3.75 seconds.
• Batch time: Tbatch = 0.03 + 3.75 = 3.78 seconds, so modelled latency is 3780 ms.
• Token throughput: 40 × 8 = 320 tokens/sec.
• Request throughput: 320 / 150 ≈ 2.13 req/sec.

For this eight-request LLM batch, the calculator reports the following metrics:

In this model, the larger LLM batch multiplies throughput by the batch size while the latency calculation remains driven by tokens per request and fixed overhead. Reducing batch size for responsiveness lowers the modelled throughput and can increase the calculated unit cost.

LLM batch inference limitations and operational considerations

This LLM batch-inference calculator is a first-order model. A deployed serving system can differ because of:

• Queueing delay: asynchronously arriving requests can wait for a batch to fill or for a dispatch timer. That wait can dominate tail latency and is not included in the calculation.
• Variable sequence lengths and padding: runtimes may pad to the longest sequence in a batch, spending compute on shorter requests. Where padding is substantial, use token counts that reflect padded lengths.
• Nonlinear scaling: token throughput can stop scaling linearly because of memory bandwidth, KV-cache pressure, kernel inefficiency, or scheduler overhead.
• Different prefill and decode rates: prompt prefill and generated-token decode can have materially different performance, which one rate R cannot show.
• Streaming and tool calls: streaming, retrieval, and tool execution can reduce batching opportunities and add latency outside the model execution.
• VRAM constraints: memory commonly caps batch size. An out-of-memory limit may require smaller batches, shorter contexts, quantization, or KV-cache optimization.

Use the batch-inference figures to explore the trade-off, then profile and load-test the actual serving configuration, including P95/P99 latency, before setting an LLM SLO.

LLM batch inference planning guidance

Batch-inference outputs become actionable when tied to an LLM serving decision. Common uses of the four metrics include:

• “Can we meet a latency target?” Compare per-request latency with the SLO. If it already exceeds the target before queueing, reduce tokens per request, raise token rate with faster hardware or kernels, or reduce overhead.
• “How many GPUs do we need?” Use request throughput as per-GPU capacity, then add headroom for bursts and tail latency rather than planning only for average arrival rate.
• “What does batching do to unit economics?” Use cost per 1,000 tokens to compare hardware configurations or internal budgets. Recheck it when low traffic reduces the realized batch size.
• “What should the maximum batch size be?” Increase B through measured benchmarks until returns diminish, memory limits appear, or the latency budget is exhausted. The largest possible batch is not necessarily the best serving setting.

A practical LLM capacity exercise compares a high-batch, low-overhead case; an expected realized batch size; and batch size 1. If the batch-size-1 case is unacceptable, consider autoscaling, request shaping, or a fallback model.

Choosing realistic LLM batch inference inputs

LLM batch-inference estimates are only as reliable as the measurements behind their inputs. Use production-like values whenever possible.

• Token rate (R): measure on the intended model, precision, and runtime. Speculative decoding, quantization, and tensor parallelism can change the effective rate substantially.
• Prompt and completion tokens: use a distribution-aware value. Customer-facing chat workloads can have P95 completions far above the median; include output caps where relevant.
• Overhead (O): include work performed once per dispatched batch, such as collation, padding, host-to-device copies, and synchronization. Replace an initial estimate with a measured value when available.
• Cost per hour (K): use the complete hardware cost relevant to the decision, such as a cloud instance rate or an on-premises amortized hardware, power, and operations cost.

To approximate padding waste in an LLM batch, inflate the prompt or completion count to the padded length used by the runtime. This is still an approximation, but it makes the token-time estimate more representative when sequence-length variation is large.

LLM batch inference FAQ

Does LLM batching always reduce request latency?

Not necessarily. In this calculator, per-request latency is largely independent of batch size because processing time is based on tokens per request and overhead is fixed. Actual LLM serving latency can rise through queueing, padding, or memory pressure. Batching is chiefly a throughput and unit-cost lever; faster kernels, fewer tokens, and better scheduling are more direct latency levers.

Why does this batch inference calculator use one token rate?

LLM servers often process prompt prefill and token decode at different rates. This calculator uses one effective rate to keep the model transparent. If separate measurements exist, convert each token portion to time and use an effective rate only as an approximation for this simplified model.

What does cost per 1,000 LLM tokens include?

It is the modeled hardware cost from the hourly price and calculated batch time, allocated across all tokens in the batch. It excludes engineering, networking, storage, and other platform expenses, so it is a hardware unit-cost baseline rather than a complete accounting model.