Retries that don't make it worse

A request failed. Try it again. It is the first thing anyone reaches for, it is three lines of code, and it genuinely does fix the most common class of production failure — the brief, unlucky, nobody's-fault kind.

It is also the easiest piece of reliability code to get wrong, because a wrong retry doesn't look like a bug. It looks like it's working.

Not every failure is worth retrying

The first question is whether trying again could possibly produce a different answer.

Retry these. A connection reset, a DNS blip, a read timeout, a 503 from a service that was mid-deploy, a 429 telling you to slow down. Nothing about the request was wrong. The world was briefly unavailable.

Don't retry these. A 401, a 403, a 422 on a malformed body, a unique-constraint violation. The request itself is the problem. Sending it again produces the same rejection, and all you've bought is latency and noise. A bad password retried five times is five failed logins in the audit log, and on some systems that's a lockout.

The boundary that matters isn't the status code, it's the question: is the input wrong, or was the moment wrong? Status codes are a decent proxy for that, which is why retry on 5xx, 408, 429, and network errors is a reasonable default. Just know that it's a proxy. A 500 from a service that will reject this payload every single time is not worth retrying, and only you know that.

The one that gets everybody

POST is where this turns from wasteful into harmful.

Your request times out. You don't know what happened. The two possibilities look identical from where you're standing: the server never got it, or the server processed it perfectly and the response died on the way back. If you retry blind and it was the second one, you've just charged the card twice.

The fix is to make the operation safe to repeat, which means the caller generates a key — a UUID, per logical operation, not per attempt — and sends the same key on every retry. The server records it against the result. A second request with a key it has seen returns the stored result instead of doing the work again.

The important part is that the key belongs to the operation, not the attempt. Generating a fresh key inside the retry loop is a very easy mistake and it buys you exactly nothing: every attempt looks new to the server, which is the situation you were trying to escape.

Pacing

Three attempts back-to-back is barely a retry. All three land inside the same outage, fail for the same reason, and the whole thing resolves in under a second having accomplished nothing.

Space them out, roughly doubling each time, and add jitter — a random fraction of the delay. Without jitter, every client that failed during the same incident retries in lockstep, and you've built a thundering herd that re-breaks the service each time it tries to come up. Jitter is what spreads the recovery out. It is one line and it is the difference between a retry policy that helps a struggling service and one that finishes it off.

If the server told you how long to wait, in a Retry-After header, use that number instead of your own. It knows more than your backoff curve does.

Stop

Every retry policy needs a limit — a maximum number of attempts, or a deadline, or both. Not because the failure will resolve, but because a job that retries forever is indistinguishable from a job that is stuck, and it holds a worker while it does it.

Give up, record the failure somewhere a human will look, and move on. A dead-letter queue is just "the pile of things that didn't work" and it is far more useful than a log line in the middle of the night. If you want one metric for this, make it the size of that pile.

The surprising part

The failures I've found hardest to catch aren't the ones where a retry was missing. They're the ones where the retry was silently succeeding on the second attempt, every single time, and had been for months.

Nothing was broken. Nothing paged anyone. There was just a dependency failing half the time and a retry quietly papering over it, right up until the day it failed twice in a row.

So count the attempt number. A retry that works is still a signal, and it's the only early warning you get.