An Inference Service Is a Scheduler: Batching, Prefill, and Prompt Caching

An application can send identical prompts to the same model and get very different latency depending on what other requests arrived at the same time.

That is not necessarily a model-quality problem. It is usually a scheduling problem.

An inference service receives requests with different prompt lengths, generation limits, priorities, cancellation rates, and authorization boundaries. It decides when to start them, which requests share hardware work, where cached state can be reused, and how much capacity each phase receives. Those decisions determine whether an AI feature feels responsive, whether expensive accelerators stay productive, and whether one noisy workflow ruins another.

The serving layer is where a model becomes a multi-user product. Treat it as a scheduler with explicit policies, not as a transparent proxy between a browser and a model API.

Batching Is a Trade, Not a Free Throughput Upgrade

Accelerators are efficient when they work on several compatible requests together. Batching lets a service combine work, which can improve tokens per second and reduce the cost of each request.

The cost is waiting. The first request in a batch may have to sit until more requests arrive, and one slow sequence can keep resources occupied while shorter requests are ready to complete.

Three common policies make the tradeoff visible:

Policy	How it works	Main benefit	Main risk
Static batching	Wait for a fixed number of requests, then run them as a group	Predictable batch size and high utilization under steady load	A request can wait indefinitely for the final slot
Dynamic batching	Start when either a size target or a maximum wait time is reached	Bounds queueing while still capturing some batch efficiency	The correct window depends on traffic and user expectations
Continuous batching	Add new work as completed sequences release capacity	Keeps active slots full despite variable output lengths	Requires a scheduler that manages state, fairness, and cancellation carefully

Static batching is often suitable for an offline workload where a delayed start does not matter. It is a poor default for an interactive chat box. Dynamic batching introduces a deadline: run as soon as enough compatible requests arrive or after a short waiting window. Continuous batching, sometimes called in-flight batching, goes further by admitting a new request whenever an active one finishes rather than waiting for the entire original group to complete.

Continuous batching is powerful because generated answers have unpredictable lengths. A user who asks for a two-line answer should not have to wait for a neighboring request that asked for a long report. But it is not merely an implementation detail. The scheduler must decide whether to prioritize older requests, short ones, premium tenants, support incidents, or retries. Every choice has fairness and cost implications.

Queueing Must Be Visible to the Product

Many systems record only model execution time. That misses the delay a user sees before the model receives work.

Track at least four stages:

1. Admission: the service accepts a request after authentication, rate limits, validation, and budget checks.
2. Queueing: the request waits for an eligible batch or capacity slot.
3. Prefill: the model processes the input context and can begin streaming a response.
4. Decode: the service generates and delivers successive output tokens until completion, cancellation, or a limit.

With those timestamps, an incident becomes diagnosable. A higher time to first token might come from a large retrieved document, too-aggressive batching, a full queue, or a cold model replica. A slow stream after the first token points to a different set of constraints, often decode capacity or memory bandwidth.

For browser and JavaScript clients, expose enough state to avoid accidental retry storms:

type StreamStatus =
  | { state: 'queued'; requestId: string }
  | { state: 'preparing'; requestId: string }
  | { state: 'streaming'; requestId: string; text: string }
  | { state: 'complete'; requestId: string; text: string }
  | {
      state: 'limited'
      requestId: string
      text: string
      reason: 'token-budget'
    }
  | { state: 'failed'; requestId: string; retryable: boolean }

The UI should let a user cancel a request that is no longer useful. Cancellation must reach the scheduler, not merely stop painting tokens in the browser. Otherwise an abandoned stream continues occupying a batch slot and spending a shared budget. Similarly, retry controls should use the request ID and a short backoff policy so repeated clicks do not create duplicate work with subtly different queue positions.

Prefill and Decode Compete for Different Resources

The two phases of generation have different resource profiles. Prefill performs a large amount of parallel computation over the input. Decode repeatedly produces the next token while reading model weights and accumulated attention state. On the same machine, a burst of large prompts can compete with ongoing streams in ways that make neither phase perform well.

One response is prefill-decode disaggregation: dedicate some workers to processing input context and others to token generation. The prefill side prepares the state needed for decoding, then transfers it to a decode worker.

This separation can help when workloads are imbalanced:

• A document-heavy assistant with long prompts may need more prefill capacity to keep time to first token within its target.
• A chat feature that generates longer answers from short prompts may need more decode capacity to keep the stream moving.
• A mixed workload needs a measured ratio, not a permanent guess. Demand changes with product features, traffic peaks, and prompt design.

The transfer has a cost. The prepared attention state must move between workers, and that path needs high-bandwidth, predictable networking. A distributed design that looks elegant but spends too much time moving intermediate state can be slower than keeping both phases together. Measure the handoff, failure behavior, and recovery path before calling the split an optimization.

Prompt Caching Is Reused Work, Not a Shortcut Around Authorization

Many requests share an initial sequence of tokens. A system prompt, a stable tool definition, an unchanged policy, a long document queried several times, or the shared beginning of a conversation can all be processed once and reused. This is usually called prompt, prefix, or context caching.

For the service, it avoids repeating prefill work. For the product, it can reduce time to first token and cost on repeated context. For users, it should be invisible except through a faster, reliable experience.

The cache key needs more than prompt text. It should distinguish at least:

• the exact model and model revision;
• tokenizer and formatting rules;
• applicable system instructions and tool definitions;
• tenant, user, document, and authorization scope when context is not public;
• any feature flag, locale, or safety policy that changes the effective prompt.

This prevents a serious category error: confusing equal text with equal permission. Two users may ask the same question about two documents with the same title, but only one may be entitled to the source material. Reuse can be safe for a public template or one user's unchanged history. It must not leak a cached representation, timing signal, or response from one protected context into another.

Retention matters too. A cache should have a clear time-to-live, eviction policy, and operational access policy. Caching can make a sensitive request persist longer than an engineer intended, particularly if tracing, debugging, or fallback systems record cache metadata alongside user identifiers.

Choose Online or Batch Work From the User Job

Not every model request needs to run while a person watches. An online request is processed after it arrives and usually prioritizes latency. A batch workflow can accept a collection of work, optimize throughput, and return results later.

The distinction should shape the product API:

• An interactive drafting assistant needs a stream, cancellation, a clear latency target, and a bounded output policy.
• A nightly classification or enrichment job needs durable submission, idempotency, progress reporting, retry limits, and a place to inspect failures.
• A hybrid feature may acknowledge a user action immediately, queue expensive work, and notify the user only when a validated result is ready.

Trying to make an offline workflow look synchronous often creates the worst of both approaches: it burns online capacity, times out in the browser, and still gives the user no dependable completion contract. Conversely, moving a genuinely interactive task to a background queue can hide a product failure behind the word "asynchronous."

Admit Requests Deliberately

The scheduler is also a control point for protection. Before a request consumes scarce inference capacity, decide whether it may proceed:

• authenticate the caller and authorize the context they supplied;
• validate structured inputs, size limits, and requested output budget;
• enforce per-user and per-tenant rate limits;
• reserve capacity for high-priority or safety-critical workflows rather than letting bulk traffic take every slot;
• attach an idempotency key where retries could cause duplicate side effects;
• record enough metadata for cost attribution without indiscriminately retaining prompt content.

These policies are not a reason to bury an AI service behind opaque queues. They are how a multi-user product prevents a long, expensive request or an automated client from turning one model endpoint into a denial-of-service mechanism.

Optimize the Queue the User Actually Experiences

Serving optimization is often described in accelerator terms: tokens per second, batch size, cache hit rate, and utilization. Those metrics matter, but they are not the entire outcome.

The product experience includes the time before anything happens, the ability to stop stale work, the fairness of queueing, the correctness of cached context, the behavior during a worker failure, and the clarity of the UI when a request cannot be admitted. A service that maximizes raw throughput while hiding those failures has optimized its own dashboard, not the application.

Build the serving layer around explicit workload classes and measurable promises. Then batching, phase separation, and cache reuse become useful tools for meeting those promises rather than clever ways to postpone the next performance incident.