Every provider suddenly wants to sell you a model
Two thirds of provider conversations this quarter involved a language model. Most of them should not have. Here is how we are deciding what to list.
Since the start of the year, a large majority of provider conversations have included a language model somewhere. Some are genuinely useful. Many are an existing product with a chat interface stapled on, and telling them apart has become a significant part of our week.
The question we ask
Does the model do work the caller could not otherwise do, or does it restate work that already existed?
A document extraction API that reads a scanned invoice and returns structured fields is doing something genuinely hard, and doing it with a model is a reasonable implementation detail. A geocoding API that now accepts "the big Tesco near the station" is solving a real ambiguity problem.
An API that returns the same data as last year, with a paragraph of prose describing it, has added tokens and latency and taken away determinism.
The three things that change for integrators
Latency goes up by an order of magnitude. A lookup that was forty milliseconds becomes two seconds. That moves it out of the request path and into a background job, which is an architectural change, not a drop-in upgrade — and providers pitching this as a free enhancement rarely mention it.
Output stops being deterministic. The same input can produce different output on Tuesday than it did on Monday. For a display string that is tolerable. For anything you compare, cache, or use as a key, it is not.
Cost per call becomes variable and input-dependent. Our per-request pricing assumes calls cost roughly the same to serve, and token-based pricing breaks that assumption in a way we are still working through.
How we are listing them
Any listing where a model is in the response path is labelled as such, prominently. Latency figures are measured the same way as everything else, with no allowance for the fact that generation is slow. Where a provider offers both a deterministic and a generative endpoint, we list them separately rather than letting the fast one advertise the slow one.
And we ask what happens when the model is unavailable. A provider whose answer is "the endpoint returns 503" is being honest. A provider who has not considered the question is not ready to be listed.
What we are not doing
We are not adding a model to Zephiel itself. The gateway's job is to be predictable, and a component whose output varies is a poor fit for the part of the stack everything else depends on.