Streaming Is Not One Latency: TTFT, TPOT, and Goodput for AI Features

A streaming answer arriving token by token creates an easy illusion: the first visible word means the system is fast.

It may not be. A product can show a cursor promptly and then emit text so slowly that a user abandons it. It can also generate a complete answer quickly after a long blank wait. Both failures are hidden when a dashboard compresses every request into one average latency number.

Inference performance is a product contract, not a single stopwatch measurement. A useful contract separates the wait before an answer begins, the pace after it begins, the share of traffic that meets the promised experience, and the resource bottleneck behind those outcomes.

That distinction matters to any team putting a model behind a web or mobile interface. It changes what the UI should show, which request patterns the backend should accept, and which optimization is worth paying for.

An Autoregressive Response Has Two Different Phases

Most text-generation systems do not create a complete response at once. They first process the request context, then repeatedly predict the next token until they stop.

The initial context-processing phase is commonly called prefill. The generation loop that follows is the decode phase. They have different workload shapes:

• Prefill has to digest the system instructions, conversation history, retrieved documents, tool results, and current user message. Its cost rises with the size of that input.
• Decode produces one next token at a time. It reuses past attention state, but repeatedly reads the model and that state from memory. Its cost rises with the length of the answer.

The user experiences these phases differently, so measure them separately:

Metric	What it measures	The product question it answers
Time to first token (TTFT)	The delay before streamed output begins	Does the product acknowledge a request soon enough to feel responsive?
Time per output token (TPOT)	The interval between generated tokens after streaming starts	Can a user comfortably read, inspect, or interrupt the answer as it arrives?
End-to-end latency	The total time from request to completed result	Is the workflow finished within its actual deadline?
Throughput	Requests or tokens processed in a time interval	How much work can the service complete at a given load?
Goodput	Throughput that satisfies the relevant service-level objective	How much useful work are users actually receiving?

TTFT is often dominated by input length, queueing, and how the service batches prefill work. TPOT is often constrained by memory bandwidth during decoding. A change that helps one can harm the other. Increasing a batch can make the hardware more efficient while delaying the first visible result. A fast first token can make an experience feel alive even when the full answer remains slow, but that is not a reason to ignore slow generation when the user needs a completed result before acting.

A Chat Cursor Is Part of the Performance Budget

Consider two interfaces for a support assistant:

• A live-chat workflow needs a short TTFT. A user has just asked a question and is waiting for an acknowledgement that the system understood it. Slow first tokens feel like a broken submit action.
• A document-summary workflow can accept a longer TTFT if the interface makes progress legible and the eventual output is delivered at a useful pace. It may be worth spending more time processing a long document before generating a short, reliable result.

The backend should not make this decision in isolation. A frontend team needs to know which phase is slow so it can provide the right state instead of a generic spinner.

type GenerationTiming = {
  queuedAt: number
  firstTokenAt?: number
  completedAt?: number
  inputTokens: number
  outputTokens: number
}

function getGenerationExperience(timing: GenerationTiming) {
  const now = timing.completedAt ?? Date.now()
  const totalMs = now - timing.queuedAt
  const ttftMs = timing.firstTokenAt
    ? timing.firstTokenAt - timing.queuedAt
    : undefined
  const decodeMs = timing.firstTokenAt ? now - timing.firstTokenAt : 0

  return { totalMs, ttftMs, decodeMs }
}

The exact telemetry shape is less important than keeping the phases visible. Record a request identifier, model version, input and output token counts, cancellation reason, queue time, and each meaningful transition. Do not put raw customer prompts into broad performance logs by default. The timing data is usually enough for an operational investigation, while the content may be sensitive.

An interface can then distinguish between a response that is queued, one that is preparing context, one that is actively streaming, one that stopped because it reached an output limit, and one that failed. Treating all five as "loading" makes users guess whether they should wait, edit the request, or retry.

Throughput Is Not the Same as a Successful User Experience

A service can report 100 requests per second and still fail the user-facing promise for most of those requests. Perhaps 70 waited too long in a queue, or their first token arrived after the chat's target deadline. Raw throughput does not reveal that.

Goodput narrows the metric to requests that meet a meaningful objective. For a conversational feature, the objective might include a first token within a target window and a completed answer before the user is likely to leave. For a nightly extraction pipeline, it might only require completion before the next business day.

This has an important operational consequence: define service-level objectives per workflow, not per model endpoint. A single endpoint may serve interactive chat, background summarization, and internal evaluations. Their acceptable queue time, answer length, cancellation behavior, and failure costs are different.

Goodput also prevents a tempting but misleading optimization. A team may raise batch size and celebrate higher tokens per second while more interactive users wait for a batch to fill. The hardware is busier, but the product is serving fewer requests within the promised experience. That is an efficiency regression, not a win.

Know Whether You Are Waiting on Computation or Memory

The phrase "GPU utilization" sounds decisive but rarely explains an inference problem on its own. A chip can appear busy while the workload is constrained by moving values through memory rather than performing arithmetic.

Two broad bottlenecks help frame the diagnosis:

• Compute-bound work spends most of its time on mathematical operations. Prefill often looks this way because it processes many input tokens and matrix operations together.
• Memory-bandwidth-bound work spends most of its time reading and writing weights or cached attention state. Token-by-token decode often looks this way because each new token must revisit a large amount of model state.

This is why a hardware comparison cannot stop at nominal operations per second. Memory capacity determines whether a model and its attention cache fit at all. Memory bandwidth affects how quickly decode can move the state it needs. Compute capability matters strongly for prefill and other matrix-heavy work. Power draw and price matter because a technically fast configuration can still be economically wrong for the workload.

Teams running their own inference can use model FLOP utilization and model bandwidth utilization to compare actual work with the hardware's achievable compute and bandwidth. The names are less important than the practice: measure the constrained resource before assuming that more accelerator capacity or a new kernel will help.

For a hosted model API, you may not see those hardware counters. You can still infer pressure from observable symptoms: TTFT rising with prompt length, TPOT worsening under concurrency, queue time growing at peak traffic, or a provider's cached-prompt behavior changing the cost and latency profile. Treat provider metrics and your own traces as complementary evidence.

Token Budgets Are Latency and Cost Budgets

Input and output tokens are not interchangeable in an autoregressive workload. A long input makes prefill heavier. A long response repeats decoding work one token at a time. Providers often price those paths differently, and self-hosted systems pay for them through different resource constraints.

That makes a maximum-output-token setting more than a guard against unusually long prose. It is a latency, cost, and capacity policy. Ask:

• What is the shortest response that lets the user complete the job?
• Is the task interactive, or can it be queued as background work?
• Can the product ask a follow-up question instead of generating an exhaustive answer from weak context?
• Are retries, candidates, tool calls, and abandoned streams included in the budget?
• What happens when the model reaches the output limit: incomplete result, resumable continuation, or a request for clarification?

The frontend should make that policy visible in the interaction. A source-backed answer that is intentionally concise is often more useful than a long stream that consumes a shared budget and forces the user to search for the actionable sentence.

Use Metrics to Choose the Next Experiment

Optimization begins with a precise complaint. "The assistant is slow" is not precise enough to decide between smaller prompts, prompt caching, batching changes, a different model, quantization, more replicas, or a different accelerator.

Start with a representative trace and ask what changed:

1. Did TTFT increase because input context, queueing, or prefill batching grew?
2. Did TPOT increase because decode is sharing memory bandwidth with more concurrent work?
3. Did total latency rise because responses became longer, a tool call was added, or retries increased?
4. Did raw throughput improve while goodput fell for an interactive user journey?
5. Is the request actually suited to an online model call, or could it be a batch job with a different completion contract?

The answers turn performance from a vague argument about faster models into an engineering decision with a measurable outcome. A well-designed inference service may still be constrained, but it should make those constraints legible to the system and honest to the person waiting for the answer.