TL;DR

Tokens per second cannot be interpreted as a complete measure of inference performance. A useful benchmark must also specify input sequence length (ISL), output sequence length (OSL), concurrency, precision, GPU count, tensor parallelism, engine version, and latency percentiles. Time to first token (TTFT) describes how quickly a response begins, while inter-token latency (ITL) describes the consistency of the generated stream. Throughput measures total system capacity; however, goodput measures how much work remains within the required latency objectives. Compare systems only when these conditions are equivalent.

You have seen the chart.

Big number. Tokens per second. Model name in bold. Maybe a GPU in the subtitle if you are lucky.

Then someone on your team asks the only question that matters: can we run this for our customers without the chat freezing at 10am?

That chart does not answer that question. Not alone.

Here is the uncomfortable bit. A system can look like it is doing 10,000 tokens per second and still be failing you. A big slice of those tokens may belong to requests that already blew past the latency your users will accept. Busy GPUs are not the same thing as happy users.

This post is the shared vocabulary we will use on GRN.CLOUD when we talk about inference. Next in the series we put our own berry AI numbers on the table. Today we only teach you how to read the label on the jar.

Grab a coffee. Conversation first. A few research papers in the background where they earned their place.

Why one “latency” number fails here

A normal REST API mostly has one latency story. Request in, response out.

LLM inference has at least three feelings for the user:

1. How long until anything starts streaming
2. How smooth the stream feels token by token
3. How long until the whole answer is done

If you squash those into one average called “latency,” you lose the plot. Throughput has the same trap. Five thousand tok/s might mean fifty people getting a good experience… or a pile of tokens generated for requests that will time out or get kicked mid way.

So we stop asking only “how fast is the GPU?” and start asking “how much useful work did we finish inside the limits we promised?”

That second question has a name. First, the two acts of every generation.

Two phases. Every generation.

Think of a request like a short story.

Act one: prefill. The model reads the whole prompt. System message, user text, retrieved chunks, tool results. When the input is long, this act gets heavy. You feel it as a blank pause before the first token.

Act two: decode. The model writes. Token after token. That is the stream in the UI. Under load, this is where most of the “tokens per second” story lives.

request arrives
        │
        ▼
   queue wait
        │
        ▼
   PREFILL  (read the prompt)
        │
        ▼
   first token     ← TTFT = queue + prefill + first decode step
        │
        ▼
   DECODE          ← ITL / TBT / TPOT = how the stream feels
        │
        ▼
   complete

Buyer line you can take into a meeting: long context taxes the front of the request. Long answers tax the stream.

Why split them so carefully? Prefill is usually compute heavy. Decode is usually memory heavy. They fight for the same GPU when you batch them together. DistServe (Zhong et al., OSDI 2024) made that conflict the center of the design: if you only chase raw throughput, you can look busy while fewer requests still meet a real latency goal. Their answer was to treat the two phases as different jobs. You do not need to copy that architecture tomorrow. You do need the mental model when someone shows you one tok/s number for everything.

Sarathi Serve (Agrawal et al., OSDI 2024) attacks the same fight from the scheduler side. A fat new prompt can stall everyone else’s stream. Their work is about keeping decode moving while new work still arrives, and about measuring capacity as the load you can sustain under TTFT and time between tokens targets, not the load where the GPU is merely hot.

Core inference metrics

The following metrics provide a consistent basis for evaluating an inference system. Each metric describes a different part of the request lifecycle; therefore, no single value should be interpreted in isolation.

ISL (input sequence length)

Input sequence length (ISL) is defined as the number of tokens supplied to the model before generation begins. It includes the system prompt, conversation history, retrieved documents, and tool outputs. A larger ISL increases the amount of work performed during the prefill phase and generally increases time to first token. Consequently, benchmark results obtained with an ISL of 128 tokens cannot be compared directly with results obtained with an ISL of 8,000 tokens, even when the model and hardware are identical.

OSL (output sequence length)

Output sequence length (OSL) is defined as the number of tokens generated by the model in response to a request. Each output token requires an additional decode step; therefore, a larger OSL increases end-to-end latency, GPU utilisation, and operating cost. Workloads that generate long reports have a substantially different performance profile from workloads that return short classifications or brief answers. A benchmark limited to 100 output tokens is not representative of an application that regularly produces 2,000-token summaries.

TTFT (time to first token)

Time to first token (TTFT) indicates the elapsed time between submitting a request and receiving the first generated token. It includes queueing time, prompt prefill, and the first decode step. TTFT is particularly important for interactive applications because it determines how long the interface remains unresponsive before generation becomes visible. Agent workflows may incur this delay repeatedly across tool-call cycles. TTFT should therefore be measured against a dedicated service-level objective rather than being absorbed into a single average latency value.

ITL / TBT / TPOT

Inter-token latency (ITL), time between tokens (TBT), and time per output token (TPOT) are related measures of the delay between successive tokens during generation. These metrics indicate the smoothness of a streamed response. Low and stable values produce a continuous output stream, whereas high or variable values create noticeable pauses. Under load, newly arriving prefill work may compete with active decode operations and increase these delays. For this reason, benchmark reports should include the distribution of inter-token delays rather than only their arithmetic mean.

However, both overly strict and overly permissive interpretations can be misleading. Wang et al. (2024) show that an average TPOT may conceal individual stalls, while a requirement that every token interval remain below a fixed threshold may overstate the effect of an isolated delay. The complete latency distribution provides a more accurate representation of streaming quality.

Throughput

Throughput is defined as the total number of output tokens generated per second across all requests in progress. It indicates the aggregate processing capacity of the system. However, throughput does not describe the latency experienced by an individual request. A valid capacity assessment must therefore report throughput together with TTFT, inter-token latency, workload shape, and concurrency.

Goodput

Goodput is defined as the amount of work completed while satisfying specified latency objectives. For example, a request may count towards goodput only if both its TTFT and inter-token latency remain within their respective limits. DistServe uses this concept to distinguish useful capacity from raw token production. Two systems may each report 10,000 tokens per second, yet provide different effective capacity if one satisfies the service-level objective for a smaller proportion of requests.

Goodput must also be calculated carefully. Wang et al. note that a system can produce a misleading result by terminating requests that have already exceeded the latency objective. Such requests disappear from subsequent measurements even though they represent failed user interactions. In this series, goodput refers to completed work measured against a stated service-level objective, with missed and abandoned requests included in the result.

Concurrency

Concurrency is defined as the number of requests being processed or waiting for processing at the same time. It should not be interpreted as the number of registered or active users. A single agent may create several concurrent requests, while many human users may generate little concurrent load if their requests are separated by long intervals. Benchmark documentation should state how concurrency was generated and whether the test used an open-loop or closed-loop traffic model.

TP (tensor parallelism)

Tensor parallelism (TP) is a method of dividing the computations and parameters of a single model across multiple GPUs. It allows a model that exceeds the memory capacity of one GPU to be served across several devices. However, the GPUs must exchange intermediate results during inference, which introduces communication overhead. Increasing the TP degree may therefore solve a memory-capacity constraint without producing a proportional improvement in throughput or latency.

Precision (for example fp16)

Precision indicates the numerical format used to represent model weights and, in some configurations, the KV cache. Examples include FP16, BF16, FP8, and INT8. Lower-precision formats reduce memory consumption and may increase throughput; however, they can also affect numerical stability or model quality. Benchmark comparisons should therefore identify the precision used and report the quality-validation method applied after quantisation.

KV cache pressure

KV cache pressure indicates the proportion of memory allocated to key-value cache data for active sequences. As utilisation approaches the available limit, new requests may be queued and active requests may be pre-empted. Pre-emption can discard partially completed decode work and increase latency when the request resumes. KV cache utilisation and pre-emption frequency are therefore useful leading indicators of capacity saturation.

For this series, a throughput figure is considered directly comparable only when the report also specifies ISL, OSL, concurrency, precision, GPU count, TP degree, inference engine and version, and the relevant latency percentiles. If these fields are absent, the result should be treated as directional rather than conclusive.

Averages lie. Percentiles tell the truth.

Mean TTFT of 230 ms can hide a p99 of several seconds. At real traffic that is a lot of angry sessions per day.

LLM serving usually shows a wider gap between typical and tail than a normal web service. Prompt lengths vary. New prefills interfere with ongoing decode. KV pressure causes preemption. Output lengths wander. Those effects stack. Sarathi’s capacity plots are basically this lesson drawn as a curve: raise the load, watch when TTFT or time between tokens stops meeting the target.

So when we publish benches, we will talk p50 / p95 / p99, not only the happy mean. If a vendor only shows the mean, you already know what they hid.

“Concurrency 32” is not “32 humans”

Bench concurrency means overlapping requests. Not people who think, click, and go make tea.

Real life: one agent can open several requests alone; batching helps until the queue grows and TTFT falls off a cliff; the interesting number is the sweet spot before latency stops being acceptable.

How you drive the bench matters too. Closed loop traffic (send the next request only after the previous finishes) can look calm when the system is slow, because it never piles work on. Open loop traffic (fixed arrival rate whether you are ready or not) is closer to production. Optimistic closed loop numbers have burned more capacity plans than we like to admit.

If a vendor only shows peak throughput at a concurrency you would never run, smile and ask for the latency curve under open style load.

Knobs that quietly rewrite the chart

You do not need our full matrix yet. You need better questions.

1. Model size. More parameters usually mean more VRAM and fewer tokens per second per GPU.
2. Tensor parallelism. Lets a large model fit. Watch communication cost.
3. Precision. Frees memory. Ask what quality bar they kept.
4. Engine and scheduler. Different stacks answer the prefill versus decode fight differently. Compare them only when ISL, OSL, concurrency, model, and precision match.
5. Path to the GPU. Bare engine versus gateway with keys, budgets, and rate limits. Measure what customers hit.
6. Memory manager health. KV occupancy and preemption rate are leading indicators, not footnotes.

How we think on GRN.CLOUD: OpenShift AI under the hood, engines like vLLM where they fit, SGLang where it earns its keep, and we size before we sell. The capacity calculator on the AI/ML page is the same mental model. Memory first. Then latency under a real shape of traffic. Useful work under an SLO over vanity tok/s.

Four shapes of work

Pick yours before you argue about speed.

Shape	Input	Output	Optimize for
Chat	medium	medium	TTFT and stream smoothness together
RAG / summarize	long	short	TTFT (prefill hurts)
Long generation	short	long	decode pace and throughput
Multi tenant API	mixed	mixed	throughput while TTFT stays acceptable at p95

Our upcoming V100 post uses one fixed shape on purpose so model comparisons stay fair. Later we twist the shape.

If production is “paste a repo, short summary,” do not buy GPUs off a long generation chart.

How we will report numbers in this series

Expect a subtitle like this under every chart:

engine · version · precision · GPUs · TP · ISL · OSL · concurrency · percentile · (goodput when we have the SLO)

Post 2 follows that for Qwen 0.5B, 7B, and 14B on V100 with vLLM in fp16.

Call us out if we skip a field. We want that bar.

Where this leaves you

Next time someone drops a tok/s chart, ask:

• What was the input length and output length?
• What concurrency, and was the load open style or closed loop?
• What precision, GPUs, TP, engine, version?
• Which percentile?
• Did they report useful work under a real TTFT / stream SLO, or only raw throughput?
• Bare engine or the path users actually call?

If those answers are fuzzy, the number is a vibe. Fine for curiosity. Thin for a purchase order.

We will bring our own numbers next, with those fields filled in. Sovereign GPUs in Europe, green power in the story, and honest cliffs where packing harder stopped helping users.

Until then: size with eyes open.

Explore the AI/ML platform and capacity calculator · Talk to an engineer if you are sizing a tenant and want a second pair of eyes.

---

Further reading

Three papers we actually leaned on for this post (not a dump of every serving paper on the shelf):

1. Zhong, Y. et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. https://arxiv.org/abs/2401.09670
2. Agrawal, A. et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. https://arxiv.org/abs/2403.02310
3. Wang, Z. et al. (2024). Revisiting SLO and Goodput Metrics in LLM Serving. https://arxiv.org/abs/2410.14257

Next in this series: measured V100 performance with vLLM (full context on the label).