Rate limits in an era of retries-by-default
Every modern HTTP client retries automatically. Several of them retry badly, and the result looks exactly like an attack.
Most HTTP clients now retry by default. That is progress — transient failures should not surface to application code. But several popular clients retry in ways that turn a small problem into a large one, and we see the results at the gateway.
The pattern
An API has a brief hiccup — a slow query, a deploy, a network blip. Requests take longer than the client's timeout. The client retries. Its retry also times out, because the original request is still running and consuming capacity. It retries again.
Now there are three times the requests, the origin is more loaded than it was, and the thing that would have resolved in four seconds takes four minutes. This is a retry storm and it is entirely self-inflicted, by software that was trying to help.
What good retry behaviour looks like
Exponential backoff with jitter. Without jitter, every client that failed at the same moment retries at the same moment, and the load arrives as a spike rather than a spread. Jitter is one line and it is the difference between recovery and oscillation.
A retry budget. A client should cap retries as a fraction of total requests — ten per cent is a reasonable ceiling. Beyond that, the correct response is to fail fast rather than to keep hoping.
Respect for Retry-After. We send it on every 429 and every 503. A client that ignores it and applies its own backoff is guessing when it has been told the answer.
And no retries on non-idempotent requests without an idempotency key. A retried POST that creates something creates it twice.
What we changed on our side
We now return 429 with Retry-After rather than 503 when the cause is load rather than failure, because clients treat those differently and the distinction is real.
We also added a per-key circuit: an account generating a sustained retry storm gets a longer Retry-After and a notification explaining what we are seeing, with the specific key and endpoint. Previously they got throttled and had to work out why.
Nearly every storm we have investigated came from a default configuration nobody chose. The fix is usually four lines in the client setup, and the people involved were always surprised, because their code never said "retry".