Yeah exactly - found the parent comment quite funny.
For instance, one of the other top comments berates Claude for terrible instruction following with regards to scientific papers, whereas this one is full of praise.
Everyone is just making up their thoughts on these models based on vibes.
Claims about Ox Alpha performing at Fable level were from the social media hype cycle. Everything new in the LLM space brings a wave of influencers hyping it up as a revolutionary leap forward. Don’t forget to like and subscribe to learn more.
It is a capable small model, but it’s not frontier level. The interesting part will be seeing the model size, how it responds to quantization, and how fast it runs on the kind of non-server hardware that we can buy without selling a kidney.
They don't have to be getting paid. The natural bias of media is towards laziness and sensationalism (stolen from Jon Stewart, so maybe the same is true about comments).
That benchmark is super sus. Until someone pointed it out, the top performing open weights model was a Kimi K3 fine tune from their sponsor (abacusai/Smaug-Agentic). Now, it's not on the list.
What is there to phish? These are simple vibe coded websites providing information for a certain topic, nothing else. In this case, that 2nd url is a website with information regarding the new model as well as a broken chat interface to try out.
You do realise there are hundreds of these types of 'sites'.
This one that is listed is designed to rank on Google as an informational source (although unofficial and not from z.ai which is why I said it is phishing)
Assuming you are technical you are able to discern this, imagine the average person.
> You do realise there are hundreds of these types of 'sites'.
Yes, and its these sorts of websites I am asking about.
> This one that is listed is designed to rank on Google as an informational source (although unofficial and not from z.ai which is why I said it is phishing)
Even if it weren't slopped together, 65% vs 80% on 10 tasks just isn't a significant difference. For 80% power to distinguish at a significance level of 0.05, you'd need more like 140 samples, if those were the true success probabilities.
The number one problem in LLM benchmarking is that people try to draw conclusions from sample sizes far too small to conclude anything but "it works sometimes, it fails sometimes, hard to say which is better." (The number two problem is that people run benchmarks blindly without checking that they measure something meaningful.)
I really want to see hard evidence of distillation before I buy into it. Seems like a lot of sour grapes over not having the sort of lead assumed. In this field, it has been shown repeatedly that leaps in performance come swiftly and without notice.
With the amount of Claudish on the internet now, and in source code repositories (how many Claudish README.mds have you seen?), you don't have to make a single API call to end up with a model that talks like Claude.
And critically, like contracts in general, Anthropic's terms of service is only binding upon the user/counterparty. So even if a company say specifically sought out 'claude-like' content, and claude code traces available on the internet, if they don't use the Anthropic platform there is no ToS claim.
Half of the new open-source stuff on github is written by claude now, all the way from issues to docs. Models are just vacuuming up this dataset during pretraining, naturally picking up the tone. You don't even need direct distillation via api anymore when the whole internet has turned into one big snapshot of Anthropic's weights
"Black-Box On-Policy Distillation of Large Language Models", Microsoft Research, https://aka.ms/GAD-project
> 'GAD consistently surpasses standard sequence-level distillation, delivering superior generalization and achieving performance that rivals the proprietary teacher. These results validate GAD as an effective and robust
solution for black-box LLM distillation.'
No RL, although I'm a little bit surprised to see MS Research publishing a paper on distilling GPT5?
That's an interesting paper, but there is virtually no discussion of reasoning behaviors or optimization for long-horizon tasks (i.e., all of the recent advances in LLMs that people care about). The evaluation methodology also is pretty dated:
> We reserve 500 samples of LMSYS-Chat-1M-Clean as the primary test set. We also include test datasets consisting of a 500-sample subset split from Dolly [6], the 252-sample SelfInst dataset [37], and the 80-question Vicuna benchmark [3] to evaluate out-of-distribution generalization. We report the GPT-4o evaluation scores [45, 10], where GPT-4o first generates reference answers and then scores the output of the student model against them. We also conduct human evaluations on the LMSYS-Chat-1M-Clean test set for qualitative assessment.
I've been having oxa and sol do architecture design then compare notes. Sol is definitely still way ahead. But there's reliably some really good wins ideas and concepts that OxA throws out there that Sol is very happy to encorporate.
One thing that I think matters a lot for the non developers, all three of us (sol, oxa, and me) usually agree that oxa's write up is far far better. It explains the situation very well, and has great structure for its write ups. Sol gets the job done, but it's terrible at re-explaining the problem for humans, at laying out information. It also doesn't show it's thinking, so it's imo a terrible peer to work with!
Kinda useless to compare simply based on model without considering harness. Different agents handle the context etc completely differently. I would like to start seeing these model vs model comparisons across different harnesses.
omp+0x-alpha beat both cc+fable and codex-sol in creating/refactoring a big eval setup. the former just knows where things should belong and completed the task all the way while the other two failed on both metrics.
Doesn't matter if the chip is 100x-1000x more efficient and faster, and you can just make a new one for new weights. Imagine being able to run GPT Sol at 1k tokens/second a year from now, at a 100x lower cost per token than now. Would that be useful? Or Qwen 3.8 27B at 10k tokens/second. The super long thinking that makes qwen so effective would take a couple of seconds.
> Imagine being able to run GPT Sol at 1k tokens/second a year from now, at a 100x lower cost per token than now. Would that be useful?
Given the current rate of change, it would be hard to guess either way. By some measures the cost at fixed quality score goes down vastly faster than that:
A similar trend is evident in the cost of models scoring above 50% on GPQA, a substantially more challenging benchmark than MMLU. There, inference costs declined from $15 per million tokens in May 2024 to $0.12 per million tokens by December 2024 (Phi 4).
Is not the same a record that may be broken in the next 100 years than one that will be broken most days till end of year. One is an event, the other is a running process with no clear end, besides that won’t be the current point.
I was following the development and it sounded like they ran out of money and had to release something. And was followed almost immediately by a 99% layoff.
I liked it, but it felt like 1/10th of a game. On the other hand they only charged about 1/10th of a game ($5). I played it for about 10 hours so I feel like I easily got my monies worth.
