SELINA.ai
Sign in
technical-depth

Latency Budgets: Where the Milliseconds Actually Go (With Real Technical-Depth)

You have a fixed number of milliseconds to respond to a user before they decide your product is broken. A latency budget is how you allocate those milliseconds across every component in the request path. This sounds straightforward until you actually instrument it and discover that the model inference you obsess over is sometimes less than half the total. The rest is network, queue time, encryption, PII scrubbing, prompt assembly, and a half-dozen other stages that nobody puts in the marketing benchmark. This piece is an attempt at real technical-depth on where time goes in a production AI system, written from the perspective of people who ship one.

Key Takeaways

What Is a Latency Budget, Exactly?

A latency budget is a fixed time envelope, defined by your product's UX requirements, that you divide across every component in the critical path of a request. If your voice assistant needs to feel conversational, you might have 400ms total. If you are serving a chat interface where users see a streaming cursor, maybe 1200ms to first token is tolerable. The budget is the constraint; architecture is how you spend it.

The concept is borrowed from distributed systems engineering, where it has been standard practice for years. What makes AI systems different is the number of stages that are variable and opaque: model queue time depends on provider load, prefill time scales with prompt length, and reasoning models can blow any fixed budget entirely.

What Actually Determines Time to First Token?

TTFT, the delay between the user pressing send and seeing the first word appear, is the number users feel most viscerally. It breaks down into at least three sequential stages:

  1. Network latency to reach the inference server, including DNS resolution and TLS handshake. A new TLS connection costs 1-3 extra round trips. If your user is in Frankfurt and your inference endpoint is in Virginia, physics alone gives you 80-90ms one way.
  2. Queue time at the provider. This ranges from zero under light load to several seconds during peak. You have no control over it on a shared API. It is the single largest source of variance in most production deployments.
  3. Prefill, where the model computes attention across every token in your prompt to build the key-value cache. Longer prompts produce higher TTFT even if the output is identical. A 500-token prompt and a 50,000-token prompt will have dramatically different prefill costs on the same model.

The first decoder step follows prefill and is usually fast relative to the other stages. But it only begins after all three preceding stages complete. They are serial. There is no parallelism to exploit here.

Where Does Infrastructure Overhead Hide?

Between your application server and the model, there is a stack of invisible middleware eating milliseconds. Gateway overhead for auth, rate limiting, and routing typically adds 10-50ms. If you are using a RAG pipeline, your retrieval step (embedding the query, hitting a vector database, reranking results) can easily consume 50-200ms before the model prompt is even assembled.

We see this in our own stack. Prompt assembly, where you merge the user's message with conversation history, retrieved context, and system instructions, is not free. It involves string operations, token counting, and truncation logic. For a system with adaptive memory (which is what we run), there is also a retrieval and relevance-filtering step against the user's stored context. That is real compute, and it happens before anything touches a frontier model.

Then there are the stages most latency explainers never mention, because most products do not have them.

How Much Latency Does Privacy Actually Cost?

This is the part we can speak to with specificity, because we build a privacy-first assistant and have to account for these costs on every request.

Privacy-specific stages that consume milliseconds include:

The common framing is that privacy is a "tax" on latency. We think that is backwards. These stages need explicit line items in your latency budget, designed in from the start. When you bolt them on as an afterthought, they blow your budget because nobody accounted for them. When you budget for them deliberately, you make different architectural choices upstream (shorter prompts, smarter retrieval, fewer round trips) that keep the total within envelope.

We do not claim this is free. It is not. But the cost is predictable and bounded if you treat it as a first-class budget item rather than a surprise.

Why Is the Decode Phase So Slow Relative to Compute Capacity?

Because it is not compute-bound. It is memory-bandwidth-bound.

During the decode phase, each new token requires reading the model's weights and the accumulated KV cache from memory. The arithmetic to produce a single token is trivial compared to the data movement required to set it up. Meta researchers have noted that people over-index on raw NPU TOPS when the actual bottleneck during decode is memory bandwidth: you have to stream the entire model's weights for every token.

This is why GPU utilization during decode drops to 20-40%. The chip is mostly waiting for memory reads to complete. Adding more compute does not help. Adding faster memory, or reducing model size (quantization), does.

This has a direct architectural consequence: prefill (processing the input) is compute-bound, but decode (generating the output) is memory-bound. They want different hardware. Which is exactly why the industry is moving toward disaggregated inference.

What Is Disaggregated Inference and Why Does It Matter for Latency?

Disaggregated inference separates the prefill and decode phases onto different, specialized hardware. GPUs handle the compute-heavy prefill. Memory-optimized chips handle the bandwidth-bound decode. SambaNova's partnership with Intel on a heterogeneous GPU/RDU/Xeon setup has shown over 50% throughput gains without a TTFT penalty.

For latency budgets, this matters because it decouples two previously linked constraints. Before disaggregation, you sized your GPU fleet for whichever phase was more demanding, and the other phase wasted resources. Separating them lets you optimize each phase independently, which tightens variance and makes your P95 closer to your P50.

How Bad Is the Gap Between P50 and P95?

Worse than you think, and it is the number that actually defines your user experience.

One 2026 benchmark found P95 latency inflates 1.6 to 3.2× over P50, with the average across tests landing around 2.1×. If your P50 TTFT is 300ms (which looks great on a dashboard), your P95 might be 630-960ms. One in twenty requests is two to three times slower than the number you would put on a slide.

This gap comes almost entirely from queue time variance on shared infrastructure. When you are calling a frontier provider's API, you are sharing capacity with every other customer. Their traffic spikes are your tail latency.

For us, this is also a trust question. When you cannot see whose multi-tenant queue is causing your worst 5% of responses, you cannot diagnose, explain, or fix the problem. You can only wait for it to pass. An architecture with fewer third-party hops gives you a tighter tail and an auditable one. We are honest that we still depend on frontier providers for inference (we route to a stack of frontier models per task), so we inherit some of this variance. But every hop we can remove from the critical path removes both a latency source and a data-exposure surface.

What Happens to Latency Budgets with Reasoning Models?

They break.

Reasoning models generate internal "thinking" tokens before producing the visible answer. These thinking tokens are real computation: the model is effectively doing chain-of-thought work that can produce vastly better outputs for complex tasks. The cost is that first-answer latency can inflate from sub-second to 10-150 seconds.

A 2026 latency report put it bluntly: reasoning mode is not a small latency increment. It is an entirely different latency category. Sub-2-second UX cannot use it.

This creates a routing decision that belongs in the latency budget. If your system can classify requests into "needs reasoning" and "does not need reasoning" before dispatching, you can route only the complex ones to a reasoning model and handle the rest with a faster path. The classification itself costs time (another budget line item), but it saves far more than it spends on average. We do this. It is not trivial to get right, and misclassification in either direction has a cost: wasted latency on easy questions, or shallow answers on hard ones.

How Tight Are Voice Application Budgets?

Tighter than almost anything else in production AI.

A conversational voice assistant has roughly 400ms from the moment a user stops speaking to the moment the assistant's voice begins. Within that budget, you need speech-to-text transcription, prompt assembly, TTFT from the model, and text-to-speech synthesis on the first sentence. The TTS step alone adds 100-200ms after the model produces its first sentence. Work backwards: if TTS startup takes 150ms and you need 50ms for STT, you have roughly 200ms left for everything else, including network, queue, prefill, and first decode.

200ms. That is your entire model budget for a voice-first product. It explains why voice assistants often feel slightly off. Any variance in queue time, any prompt that runs long, and you cross the threshold where the pause feels unnatural.

Does On-Device Inference Actually Solve This?

For certain workloads, yes. It eliminates network RTT, queue time, and TLS overhead in one architectural move. On-device inference has crossed a usability threshold in 2026, with sub-20ms latency demonstrated for production computer vision models on mobile hardware.

For LLMs, the picture is more constrained. On-device models are smaller (memory-limited by the device), which means less capable on complex reasoning tasks. But for classification, short-form generation, and intent detection, they are fast enough and predictable enough to serve as a first pass, with cloud fallback for requests that need a larger model.

The latency advantage is substantial. Cloud round-trips characteristically add 200-500ms before a user sees the first token. On-device processing eliminates that entire block. For applications like AR overlays, live translation, or voice assistants, that 200-500ms is the difference between usable and not.

The privacy co-benefit is real and not incidental. The shift toward on-device AI is described in 2026 as a structural change driven jointly by privacy regulations (GDPR, the EU AI Act) and rising cloud computing costs. Data that never leaves the device cannot be breached in transit, cannot be logged on a third-party server, and does not require a network hop. Privacy and latency are the same architectural decision, not a trade-off.

How Does European Data Residency Affect Latency?

European data residency is treated as a hard requirement for enterprise AI, and it compounds latency in a way that most US-centric benchmarks obscure. If your inference must happen in an EU region (for GDPR compliance or contractual reasons), and the closest available GPU capacity is in a different EU availability zone, you pick up cross-region latency that a US-based benchmark would never show.

Routing inference traffic through US-based platforms both violates GDPR and introduces additional network latency. This is a rare case where compliance and performance point the same direction: serving from EU infrastructure is both legally required and faster for EU users. But EU GPU capacity is thinner than US capacity, which means queue times can be longer. Your P95 in Frankfurt may be worse than your P95 in Virginia even with better network latency, because there are fewer GPUs absorbing load spikes.

For our EU users, this is a real constraint we navigate. We cannot simply route everything to the cheapest or fastest global endpoint. Data residency is a hard line item in the latency budget, not a soft preference.

What Should an SLO for TTFT Look Like?

A May 2026 piece on SLO engineering makes the point cleanly: without an explicit TTFT service-level objective, you have no target to size capacity against, no threshold to trigger autoscaling, and no error budget to signal trouble. The fix for latency variance is not "add more GPUs." It is establishing a target and building the feedback loops to maintain it.

A practical SLO has three components:

If you are building on third-party inference APIs, your SLO is partially outside your control. You can mitigate this with provider redundancy (route to a second provider when the primary exceeds your TTFT threshold), but that adds complexity and its own latency for the failover logic. There is no free lunch.

How Do You Build a Latency Budget in Practice?

Start from the user experience and work backwards. Here is a skeleton for a chat-style AI product with a 1200ms TTFT target at P95:

Sum the worst cases and you get roughly 1150ms, which is cutting it close against a 1200ms P95 target. Sum the best cases and you get about 155ms. The spread is 7× between best and worst. This is why budgets matter: if you do not instrument each stage independently, you cannot tell which one is blowing up when your SLO trips.

Every product will have a different breakdown. Voice products have tighter total budgets. Products with long system prompts or extensive memory will have larger prefill costs. Products doing retrieval-augmented generation have an entire retrieval pipeline to account for. The point is not the specific numbers above. The point is that each stage is an explicit, measured, budgeted line item.

What Do We Get Wrong?

We are not going to pretend we have solved all of this. A few honest limitations:

We still depend on frontier providers for inference, which means we inherit their queue-time variance. Our memory is encrypted at rest but is NOT end-to-end encrypted, because a slice of each request must reach a frontier provider. We keep non-content operational metadata for a short retention window, not zero retention. Our account system is protected, not encrypted in the way our content storage is.

Our adaptive memory system adds latency that a stateless chatbot does not have. Retrieving, filtering, and injecting relevant memory into a prompt is real work. We think the trade-off is correct (a personalized response is worth more than a generic fast one), but we are not going to claim the latency is zero.

And reasoning-mode routing is still imperfect. Sometimes a request that could have been handled by a fast model gets sent to a reasoning path, and the user waits longer than they needed to. Sometimes the reverse happens, and a complex question gets a shallow answer quickly. We are iterating on this. It is a classification problem with real costs in both directions.

The Budget Is the Architecture

If you take one thing from this piece, let it be this: your latency budget is not a document you write after you have built the system. It is the system. Every architectural decision, which model to call, whether to retrieve context, where to run inference, how to handle privacy stages, is a decision about how to spend a fixed number of milliseconds. The budget is not describing the architecture. It is the architecture, expressed in time.

Instrument every stage. Set an SLO at P95. Build the feedback loops to know when you are overspending. And put your privacy stages in the budget as first-class citizens, because bolting them on after the fact is how you end up 300ms over target with no idea where the time went.

If you want to see how this works in a product that actually ships with these constraints, start a free 7-day trial, no card required.

Frequently Asked Questions

What is a latency budget?

It's a fixed time envelope, set by a product's UX requirements, that gets divided across every component in a request's critical path, such as network, queue time, prefill, and decode. The budget acts as the constraint, and system architecture determines how that time is spent.

What determines Time to First Token (TTFT)?

TTFT is governed by at least three sequential stages: network latency (including DNS and TLS handshake), queue time at the provider (which varies from zero to several seconds), and prefill (computing attention across the prompt to build the KV cache). These stages are serial, so there's no parallelism to speed things up.

Why does privacy add latency, and how should it be handled?

Privacy-specific stages like PII redaction, encryption/decryption of stored context, and consent/policy checks all consume real milliseconds in the critical path. The article argues these costs are predictable and bounded if explicitly budgeted for from the start, rather than treated as an afterthought that blows the budget.

Why is the decode phase slow even on powerful hardware?

The decode phase is memory-bandwidth-bound rather than compute-bound, since each new token requires streaming the model's weights and KV cache from memory. This is why GPU utilization during decode drops to 20-40%, and why adding more compute (TOPS) doesn't help unless memory bandwidth or model size is addressed.

Why should teams budget against P95 instead of P50 latency?

A 2026 benchmark found P95 latency inflates 1.6 to 3.2× over P50, averaging about 2.1×, mostly due to queue time variance on shared infrastructure. This means one in twenty requests can be two to three times slower than the P50 number typically shown on dashboards, so budgeting only for P50 misrepresents real user experience.

Sources & References

Michael C.

Michael C.

Founder & Principal Engineer, Selina Labs

Michael builds Selina, a privacy-first AI that remembers you across conversations. He ships security-sensitive AI in production — real attacks, real fixes, measured in minutes and dollars — and writes about privacy, security, and LLMs from that seat. Top Rated Plus and expert-verified on Upwork.

Learn more about Selina.ai