There’s no shortage of posts defining what an SLA is. Most of them stop at defining a number like “99.9% availability.” How you actually build that number, and how you hold it, gets far less attention.
Working on low-latency services, where response speed is the product itself, you arrive at that question naturally. Here the SLA stops being an availability (up/down) problem and becomes a latency problem. The server can be alive and the response can still blow past its budget, and that’s already a failure. So an SLA has to be handled along two axes. Design builds the latency budget; operation holds it.
flowchart LR
subgraph design["Built by design"]
D1[Timeout budget]
D2[Hot-path caching]
D3[Graceful degradation]
D4[Fault isolation]
D5[Load shedding]
end
subgraph ops["Held by operation"]
O1[p99 SLO]
O2[Error budget burn rate]
O3[Capacity headroom]
O4[Deploy latency gate]
O5[Error budget review]
end
design --> SLA[(Latency SLA)]
ops --> SLA
SLA, SLO, SLI, error budget
The four terms form a single chain.
An SLI (Service Level Indicator) is what you actually measure, like a request’s latency or its error rate.
An SLO (Service Level Objective) is the target you set on that indicator, such as “keep p99 latency under 200ms.”
An SLA (Service Level Agreement) is that target promised externally, with consequences when you break it.
The error budget is the SLO inverted: if the SLO is 99.9%, the remaining 0.1% is your allowed failure, a budget you’re free to spend.
This chain matters because the error budget becomes the currency of decisions. While the budget holds, you can push deploys; when it runs dry, you have to spend on reliability. Beyond being a target, the SLO becomes the line that defines the balance in that budget.
What changes for a low-latency service is the SLI. An availability-centered SLA sets the SLI to “fraction of successful responses,” but a low-latency service sets it to a latency percentile. The indicator moves from “how often were we alive” to “how often were we fast enough.” That single shift reshapes every design and operational choice that follows.
Three hard parts of a latency SLA
First, tail latency amplifies as it composes. When one request fans out to several downstream services, the overall response is hostage to the slowest of them. Even if each downstream has a decent p99, waiting on several at once raises the odds that one of them lands on its p99. A slowness that’s rare in isolation shows up often once it accumulates across requests.
Second, traffic surges create a positive feedback loop. When requests pile up, latency rises; as latency rises, connections and threads stay held longer; as they stay held, throughput drops and the queue grows further. Autoscaling helps, but new instances take time to come up. In that gap an already-slow service gets slower, and slowness feeds more slowness until the surge window stretches out.
Third, a latency SLO violation recovers differently from an availability one. When a server dies and comes back, availability recovers instantly; latency does not. Requests backed up during a surge remain as a backlog, still burning budget for a while even after traffic returns to normal. That makes latency SLAs slow to respond to. By the time you react after it breaks, much of the budget is already spent. Without building the budget up front by design, there’s no balance left for operation to hold.
The way design builds a latency budget converges on a single attitude. Rather than waiting to give a complete answer, give what you can within budget and degrade the rest.
Timeout budget
Designing a low-latency service starts by fixing the overall response budget. Once you’ve decided “this request answers within 200ms,” that budget has to be divided among the downstream calls. This is the timeout budget.
The key is that a timeout isn’t a static constant; it’s derived from the budget that remains. If upstream has already spent 120ms, only 80ms is left for the downstream call. Passing that remaining budget down the call chain is deadline propagation. Each stage knows how much is left until the overall deadline and only attempts within it.
Skip this and it breaks in two directions. If the upstream timeout is shorter than the downstream one, the downstream is still working when upstream gives up, and that work is wasted wholesale. If the upstream is longer, then even when the downstream fails, upstream waits it out and overshoots the total budget. A timeout is closer to the rule for dividing the budget than to a mere safety net.
Hot-path caching
The most direct way to build a latency budget is caching. A cache hit skips a downstream call wholesale, and what you skip is budget you’ve won back. In a low-latency service, caching is less a performance optimization than a precondition for the SLO to hold at all.
So the design question shifts from whether to cache to what you can let go stale, and by how much. The invalidation interval and the tolerance for staleness are themselves an SLO negotiation. Insist on perfectly fresh data and the hit rate falls and the budget vanishes. Allow stale values within some bound and budget appears. Low-latency services usually choose the latter, judging that a little staleness beats a slow correct answer.
Graceful degradation
When a complete response can’t be built within budget, a low-latency service chooses a degraded response over a failure. If personalized recommendations don’t arrive in time, it serves a default list; if supplementary fields can’t be filled, it responds with the essentials. The user sees a slightly less rich screen, but sees it fast.
This isn’t exception handling; it has to be designed as part of the normal response path. Decide up front that extras get dropped when they’re late, and at the moment the budget is about to blow, the system lowers the completeness of the response on its own to stay within budget. When completeness and speed collide, a low-latency service protects speed and yields completeness. That priority has to be set at design time rather than scattered through the code.
Fault isolation
Isolation is building walls so one dependency’s slowness can’t eat the whole budget. Two patterns form its axis. A bulkhead separates resource pools (connections, threads) per dependency, so if one side slows down and exhausts its pool, the other paths keep flowing. Like a ship’s compartments, one floods without sinking the whole vessel.
A circuit breaker cuts off calls to a dependency that keeps failing or running slow. Instead of burning budget calling a downstream that plainly can’t answer, it opens the circuit, fails fast, and hands off to a fallback. Without isolation, one slow dependency drags the whole system down in a cascade, and that’s usually where the slowness-feeds-slowness loop from earlier begins.
Load shedding
When traffic surges beyond what the budget can hold, a low-latency service drops the excess fast. This is load shedding. It sounds paradoxical: a service built to serve requests refuses them. But the alternative is worse. Accept everyone and queue them, and the longer the queue grows the slower every response gets, until all of them overshoot the budget.
The core judgment is this: for a request that can’t be served within budget anyway, rejecting it early beats failing it late. Reject early and that capacity can hold the SLO for the rest. Rather than letting the queue grow without bound, set the boundary of what can be accepted and turn away anything past it immediately. Load shedding is close to the most decisive form of degradation: instead of lowering a response’s completeness, it gives up the response entirely to protect the system as a whole.
With the budget built by design, operation now holds it.
The p99 SLO
Operation starts by setting the SLO on the right indicator. In a low-latency service, mean latency lies. Even if the mean is 50ms, if 1% of requests take 2 seconds, the service is slow to the users who hit that 1%, and the mean never shows that tail.
So the SLO is set on a percentile. p99 means 99% of requests came in under that value; p999 means 99.9%. The larger the traffic, the larger the absolute number of users caught in that tail, so bigger low-latency services look past p99 to p999. Defining the SLO as a percentile, “p99 latency at or under 200ms,” is what makes the indicator carry the experience users actually have.
Error budget burn rate
With an SLO set, getting alerted after a violation fires is too late. Operation watches how fast the error budget is shrinking. This is the burn rate. If the budget is burning several times faster than normal, that’s a signal you’ll violate soon, even if you aren’t in violation right now.
This view turns monitoring from reactive to preemptive. Instead of asking whether the SLO is violated now, it asks when the budget runs dry at this rate. A steep burn calls for a short window and an immediate response; a gentle one calls for a long window and trend management. The trigger for an alert becomes the rate at which the balance is draining, not the balance itself.
Capacity headroom
Earlier we saw the gap where autoscaling can’t keep up with a surge. Operation absorbs that gap with headroom. Run utilization near 100% all the time and there’s no slack, so a small spike in traffic pushes latency past budget at once. In a low-latency service, running full to the edge means a small swing collapses the SLO.
So a fixed amount of spare capacity is always kept free. That slack is the buffer while autoscaling brings new capacity up. When a surge begins, the headroom takes it first, and scaling catches up in the meantime. Headroom looks like waste, but it’s closer to a premium paid to hold the latency SLO.
Deploy latency gate
Latency regressions mostly come in through deploys. New code makes one more downstream call, or serialization gets heavier, or a cache key shifts subtly and the hit rate drops. The feature works fine, and only p99 quietly gets worse. Operation blocks that regression at the deploy stage.
Send a slice of traffic to the new version as a canary, and compare that slice’s p99 against the current version. If latency gets meaningfully worse, a gate stops the deploy. It treats a latency regression exactly like a functional bug, giving it the power to block a release. Without this gate, latency degrades a little with every deploy, and unwinding that accumulation later is hard.
The error budget review cycle
The last piece is the cycle that turns the budget into policy. While the error budget holds, the team ships features fast, since there’s budget to spend if something fails. When the budget runs dry, new features stop and reliability gets the investment. The budget becomes an automatic regulator between development speed and reliability.
When this cycle runs, reliability discussions rest on a balance rather than on feeling or intuition. “Things feel shaky lately” turns into “we’ve already burned this quarter’s budget.” The budget defined by the SLO comes back, through the review, as the basis for setting the next priority.
Built by design, held by operation
Looking back, the two axes don’t separate. However tightly design builds the budget, if operation isn’t watching, a single deploy leaks it away. And however keenly operation watches, if design never built the budget in the first place, there’s no balance left to hold.
For a low-latency service, an SLA wasn’t a number written into a contract. It was the sum of the design decisions that divide the timeout, place the cache, and settle the degradation, and the operational habits that watch p99, measure the burn rate, and gate the deploy. The number was only the result; what built and held it were these decisions.