At Lorikeet, we rely heavily on inference providers – OpenAI, Anthropic and open-weight model providers. They often offer lower reliability guarantees than we want, so our product needs to stay up when one goes down.

For each prompt, we choose fallback providers based on intelligence, speed, region and cost. We might try OpenAI’s Sol 5.6, then Anthropic’s Opus 4.8, then Google’s Gemini 3.1, creating a fallback “ladder”.

Initially, we used manual feature flags to fail away from a provider experiencing an outage. But that limited our response speed to a human’s. We upgraded to a rolling, weighted error rate that increasingly shifted traffic away from degraded model/provider combinations. This state started process-local, then became shared across the fleet.

That worked reasonably well, but provider failures rarely fail fast for us. Instead of a quick 500, we often wait 15 or 30 seconds for junk or a 5xx. Every request sent to a failing provider adds customer-facing latency.

So we introduced fleet-wide circuit breakers. They deliberately use different thresholds for deciding that a system is unhealthy and deciding that it has recovered. A breaker opens when enough failures occur within a time window, stopping ordinary traffic to that model/provider. After a jittered cooldown, it becomes half-open and admits a few requests. Enough successes close it, while too many failures reopen it.

What about the worst case? If health checks rule out every otherwise permitted rung and there is still time, we make one deterministic last-resort attempt. A chance of answering is better than failing without trying.

Once an outage is detected, this limits the number of tickets that experience the provider’s slower behaviour. But we also wanted to protect requests before an outage was certain.

Inference is a good candidate for speculative execution. Before we receive a response, the request hasn’t changed durable ticket state in our system. We can therefore make hedged requests – when a request crosses a configured latency threshold, we start another against the next eligible rung and race the pair. We take the first successful response and signal cancellation to the loser.

This lets us spend more on inference to “save” tail latency for customers experiencing the worst performance. It also starts recovery before we’re even certain that a provider is down.

The implementation was fiddly, but it has proved resilient to provider outages. It’s one of the first times I’ve seen speculative execution meaningfully improve customer experience at the application layer, so I thought it was worth sharing.

Interested in how streaming responses complicate this? Send me an email. I’m excited by the infrastructure problems we get to solve and by applying classical infra tools in a novel way to adapt to our rapidly changing technology environment.