I do apologize in advance for putting you in spot like this, but there are two things I have been quite curious about.
1. Since 60% of the benchmarks contributing to AA are now private only, how do you guys ensure transparency? More precisely, how can someone verify the transparency in scoring?
2. There are also disclosures about custom benchmarking services offered to AI labs, what does AA do to keep the custom benchmarking services not getting access to the same/similar data that is used in the private benchmarks?
UAE seems to like to diversify into money losing yet culturally relevant investments like man city.
I guess if there's even a 1% chance that sama is right and money will have no value in the future (or that no company besides AI companies will have value), it's worth getting a stake in OpenAi etc at any price, especially if you have more cash than you can possibly use otherwise.
Amazing, UAE is paying protection money. The Iran war pays off for Josh Kushner, who is invested in OpenAI (and already had been "lucky" with the outrageous price that Musk paid for Cursor).
That’s completely normal in many countries, Germany for example doesn’t let you pick offensive or clearly-bad names for the child. You can’t call your kid Hässlich (Ugly) for example.
You can also not use X12-Whatever-Allowed-In-US.
But apart from that, there’s no list, you can invent names.
“If the first name you want for your child is not on the list of approved names, you can apply to have it approved. You do this when you complete the application, where you are asked to explain why you want a non-approved first name.”
I think that’s a bit clearer than getting a veto, as vetos without a whitelist can get applied inconsistently.
Some European states do that, or place restrictions.
In here, for example, the name needs approval to not be harmful to the kid, nor degrading, nor diminutive form of name can be used. Also typos are checked for.
I don't think that's unusual. In my country there isn't a fixed official list, but the law says you need to use an actual name (it can be a foreign one). If it seems too crazy it might be denied.
People here find it very strange (insane even) that in US you can name your child Apple, North West or X Æ A-Xii.
If you give it some thought, it becomes obvious that since any name is used in many public settings, society/state has a legitimate case to prevent offensive naming. That is different from state literally creating an allowlist. The huge middleground is benign names that are in no way offensive but still require you to get an approval from the state. I think this is an overreach, ymmv
In order to lighten the administrative burden that occurs when society/state has a legitimate case to prevent offensive naming, society is likely to create an approve list.
Denmark did use to have "These and only these names" and they were written into the law (yes, there are pages and pages of Danish legislation that are just names), but now any non-offensive name is okay, and for administrative purposes, an "approved"-list is kept for easy reference to things that have already been, well, approved.
Conceptually very interesting and revealing, but I found it frustrating being forced to binarize the outcome without sufficient context (ill-posed). A lot of those questions have both valid yes and no answers, depending on the context.
This is such a good use of tech! Although I have heard in the past that youtube is quick to ban any use of its content on a third party site. I hope I am wrong and this stays.
As of their latest update (AA's), I find their index hard to trust, and a bit shady. 60% of the benchmarks contributing to the AA index are now private, nobody can replicate them. And they explicitly say that they offer benchmark consulting to labs. This is a serious conflict of interest.
The latest update knocked down only openweights models from the leaderboard.
> I'm all ears. Our intention to make things clear and simple, even if we miss the mark sometimes.
Thanks. I think the very response you wrote here is a better form of marketing for the software engineers than what openAI puts out there. When everything is groundbreaking and everything tops charts, nothing is.
I suspect a lot of target demographic (developers) prefer simpler and honest communication about not just the models high points but low points too. With the linked Terra post, if I saw 1 chart that clearly highlighted the usecases for which Terra is _worse_ than higher tier models, I would be more convinced overall.
Everything trying to occupy the top spot in my brain's limited "top model" representation space is not helpful.
While I generally agree that it'd be nice to just have "here is the Pareto optimal model for your use case. Here is the specifics we optimized each model for" so users don't have to do benchmarking and research as much, they're getting much better at explaining. It's basically what the commenter above said.
The website linked starts out with:
> We’re introducing GPT-6.1 Terra, a fast, efficient model that combines detailed visual reasoning and precise computer use alongside clearer, more natural writing. A significant step up from GPT‑5.6 Terra, it excels at the communication and computer-based tasks that fill the workday, from drafting proposals and interpreting technical drawings to filling out forms and editing videos.
If you go father down, it has graphs and details with specifics of this, including:
> We measured GPT-6.1 Terra’s computer-use capabilities... At medium reasoning effort, GPT-6.1 Terra scores 15 percentage points higher than GPT-5.6 Terra at about one-third the cost. It even performs on par with GPT-6 Astra at only one-tenth its price per token.
and
> On MMMU-Pro... It even nearly matches the scores of our most capable frontier model, GPT-6 Astra, and GPT-6.1 Sol at much lower cost.
It sounds like it is trained on a lot of podcasts, judging by a quick few second skims over their demo tracks.
Also, the trend of posting on HN with much more sensationalist title first, getting to the front page, then getting the title adjusted by a moderator should probably be penalized; how is 'most authentic ever' not a marketing term? (not just this post, but a bunch of posts recently got to frontpage by making unreasonable claims)
It also seems like it trained on dialog in plays and novels. Some of these responses seem way too "perfect", like a response you would hear in a dramatic soap opera. Honestly it might be using that too. Just my take from listening to first minute or two of dialog.
Telephone conversations are quite different from in person speech. The lack of visual cues from the face, or the lag in video chat, the echo cancellation and the quantisation and comprehension, all reduce the quality subtly.
Now the Stasi archives, that would be interesting. Sometimes people suspected they were being surveilled of course.
I do apologize in advance for putting you in spot like this, but there are two things I have been quite curious about.
1. Since 60% of the benchmarks contributing to AA are now private only, how do you guys ensure transparency? More precisely, how can someone verify the transparency in scoring?
2. There are also disclosures about custom benchmarking services offered to AI labs, what does AA do to keep the custom benchmarking services not getting access to the same/similar data that is used in the private benchmarks?
reply