Streaming Is Not One Latency: TTFT, TPOT, and Goodput for AI Features
An AI response can feel fast, stall after the first word, or meet an average latency target while failing real users. Design inference around time to first token, time per output token, goodput, and the bottleneck that actually limits the workload.