Zephiel API
Engineering6 December 20167 min read

What a year in one region taught us

We ran the whole platform from a single region on purpose. Here is what broke, what did not, and why we are finally moving.

The gateway has run from a single region since we started. That was a deliberate decision and, for a year, the right one. It is ending, so this is a good moment to write down what it actually cost.

Why one region

Multi-region is not a checkbox. It is a data consistency problem, a deployment problem, a debugging problem, and a bill. Doing it early would have meant solving all four while we still did not know whether anyone wanted the product.

Running in one place meant one set of logs, one database, one deploy, and one place to look when something was wrong. For a team of four, that focus was worth more than the availability we gave up.

What actually broke

Not the region. In twelve months we had no failure attributable to running in a single place.

What broke was latency for anyone far away. A customer in Singapore was paying two hundred milliseconds of round trip before we did any work at all. Our published median was honest for European traffic and quietly misleading for everyone else, which is the kind of thing that erodes trust slowly and then all at once.

What we changed first

Before adding regions, we made the latency figures location-aware. The number on a listing is now measured from the caller's side, not ours, and we show it broken down rather than as one global median. That was a smaller piece of work than a second region and fixed the more urgent problem, which was that we were telling people something untrue.

Where we go next

Read traffic moves to a second region in the new year. Writes stay in one place until we have a good answer for consistency, and we would rather have a slow write than an ambiguous one.

The lesson is not that single-region is fine. It is that the failure we spent a year worrying about was not the failure we got.

Keep reading