Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.
> We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them.
I'm glad to hear that y'all are doing this, as I was unaware that this was something OpenRouter does. I was surprised and disappointed that there are so many problematic providers that it seems like community best practice [1] is to ban somewhere in the realm of 5-6 providers. Would it be possible to provide some way to express an even stronger preference for high quality providers? E.g. "only route to first party for this model" or, "cost, but don't route to providers that more than x% worse than the first party". I'm sure something like that can be done via the API but I haven't found a UI way to do it - and having it in the UI would go a long way towards feeling like OpenRouter is looking out for me/helping solve the problem as opposed to leaving it to me to have to figure out.
toven from openrouter, leading the team working with our providers - very interested in some of the things found in the report, I dug in to the image failures specifically, and in that case we have data showing deepinfra was correctly parsing images when the endpoint went live in july, but today fails those tests. we'll work on testing images and reasoning effort etc running constantly as chris mentions in point 3.
Hmm, let me check. That certainly seems wrong. Can you send me an email w/ your email so I can look into it? im cc at openrouter.ai. Or DM me on X? x.com/cclark
My ticket number is #107156 (opened 9 days ago). The point is that the Discord is full of people not getting responses, and I'm not even the one waiting the longest. The only help they get is from the AI assistant in the server saying don't bump the ticket or you'll be punished by getting moved to the back of the queue. So there's nothing anybody can do but wait who knows how long.
Yeah we are very under-staffed. We are hiring aggressively. I readily admit we have inadequate support staffing right now, but we are going as fast as we can to ramp up.
No stress! My first reaction to this was OMG THIS IS AN INCREDIBLE RESOURCE!! We are 100% grateful for this sort of feedback! Here is a direct quote of what I said at 8:17am this morning when someone sent me the article and I scanned it:
this is amazing!!
[8:19 AM]The first obvious win is routing around providers that arent handling image inputs correctly. That should be straightforward
[8:19 AM]The effort param stuff...I thought we had addressed that, but will dig in. This is incredible feedback
[8:20 AM]We should hire this guy.
I have attempted to report numerous errors in your Chat UI, but you seem to have decided that customer input is undesired, as you have an AI support system that ignores my problem and refuses to escalate to anyone or create a ticket.
To this day, your in-chat “Report an Issue” button still does not work consistently, and I am still billed for empty responses from many image providers.
Pulled some data - a surprisingly large number of requests to models that generate images, where the user seems to want an image, do NOT return an image. In our chatroom it's ~11%! However, in almost all of those cases, the model is instead deciding to return text, and you are being billed (correctly) for that text. We are not billing you for an image that wasn't returned.
Concretely, we pulled data on the last few days of image gen requests in our chatroom (50,893 requests). 5,846 got a text response (which is frustrating, I'm sure) and were billed for text appropriately. The model did not generate an image.
There were 16 requests where a customer was billed, but neither an image or text was returned. Those should not have been charged, and we'll see if we can either fix that issue or ensure that customers aren't charged.
One thing to note about the first graph: nobody is doing as well as the first part on tool calling, and it's not close.
This might be the fault of the other providers, but it's probably just something slightly different that the first party does with the model inference program than anybody else, and that's not sure to weights it's due to vLLM twiddling (or whatever) and probably becuase the first part actually uses their own customized inference program rather than the standard methods that all the third party providers use. This isn't nefarious, it's just the challenge of these sorts of stochastic systems.
Having been in science for decades now, and seen benchmarking across many different fields, these results are completely expected for me. LLM serving is not mechanical, it's hard to get right and has lots of unknown footguns. Even something as extreme as scrambling a matrix will still likely get results that are nearly as good as normal, and if there's a bug deep in vLLM or the tensors metadata that results in that, then it's going to be pretty hard to find unless you're an active researcher with knowledge of the particular model you're running inference on. I kind of doubt that's happening here, but maybe!
In the scientific literature, when benchmarking methods, everybody's own method performs best in their own hands. Some attribute it to researchers gaming benchmarking for publication purposes, but I think it's just what we see here: the people who made a method are just the best at using it because they know all the quirks and use it best.
Programmers are not used to thinking with that nuance, and jump to conclusions about lying about quantizations, etc., but this is really just an unavoidable part of AI/ML methods: when things aren't perfect they're still pretty good and it's going to take the model creator to truly debug it. At least until the open weights ecosystem gets a lot better at ensuring reproducibility, and model cards are nowhere detailed enough for that to happen yet.
This matches what I found experimentally with gpt-oss-20b during OpenAI's red-teaming challenge. After moving from hosted inference to running the model myself on rented H100s via vast.ai, I saw the model refuse the same kinds of prompts at noticeably different rates depending on the inference stack — differences of roughly 5–10 percentage points with otherwise identical experimental parameters and seeds.
So I very much agree that this isn't necessarily about providers secretly changing the weights. For reproducible work, the serving stack - engine, version, hardware, configuration, and probably more - really belongs in the methodology alongside the model itself.
I wrote up the results here: “In AI Sweet Harmony” (arXiv:2510.01259).
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way. (And if this was a false positive, I apologize!)
Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).
Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:
1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.
2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.
These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.
Some quick thoughts on the article itself:
1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!
2. That is bad and we will fix it. Sorry.
3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".
4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.
5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto
6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...
7. Will investigate.
8. We attempt to heal these, but obviously missed some. Will fix.
9. We do not rate limit by IP. Would love some more information here, as that is very surprising.
10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.