At least Anand has been in the business for like 30 years. I didn't know anything about him but I assumed he was a 25 year old from Stanford that VCs decided to shower money on
The classification performance remains to be seen, but presumably we'll soon start to see classification benchmarks.
For other tasks like summaries (another suggested usage) it's good to see Haiku and Luna now competing against each other on cost.
I'd love to know how the business automation market breaks down by volume of call type though - hard to imagine that decision making (e.g. branching, triage) isn't a very large part of it, greater than these other suggested Haiku use cases.
I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot
I was firmly wanting to move towards local llm, and since the Sonnet 5.5 and recent Opus speed increases, I'm putting that on hold until local inference speeds can be improved. Not sure how much juice there is to squeeze there, but I'm hopeful.
I just tried to play around with it on my RTX pro 6000 setup, it spends way too many tokens on overthinking stuff even if it’s able to catch the correct approach
Its speed is pretty good on the other hand with only 3B active parameters I am getting around 170 tkn/s on fp8
Nothing flies anywhere, we use superior high speed trains. I faxed a Humorgenehmigungsantragantrag to our Bundesunterhaltungsministerium analyst, immediately. Laugh now, but you are merely lucky the reply got delayed. As soon as the leaves on the rails are dealt with, you will get a formal response that has washed itself! Let us see who is laughing then. It is nobody!
But, most importantly, is Bundesunterhaltungsministerium a single token in Kolibri? This could make a huge difference for the model's performance in German.
Silly you, thinking the German economy is merit-based. The only question is, if the bribe fits a single token of appreciation, to win a tailored call for bids!
petition to make this thread the official german jokes thread for this post, it's kind of annoying how everyone feels the need to make a top level comment for their oh-so-great "germans suck" joke.
No, I am just standing my (and my nations) ground against what is simply bad faith disrespect and what could be described as casual racism (or at least spiritually equivalent to that, to not get into the racism definition debate).
You do not get to invalidate that.
People need to stop insulting my nation.
__
If the people you're joking with aren't laughing, you're not joking with them but about them, which is generally seen as a bad move, and something that we actually wanted to leave behind with all the social progress and all.
__
This really only happens because people (as all bullies) think that they would be free to do so, because the victims would not fight back.
(Un)fortunately though, the world now has bigger problems with genuine fascists, so neurotically cowering in fear and performatively saying "yes, punch me harder. Insult me more" can be stopped now.
__
I mean look at it. By now, it's at least better, but the first 30 comments or so were "haha germany bad haaa".
That's a terrible showing for the platform even if you're not german. What value was offered? Who would want this.
We don't even need to talk about nations and identities to see that that was just noise.
I think maybe they have enough compute to try out all the promising directions at a smaller scale first and then port them to larger run
They also have incredibly smart people that can understand which improvements and directions are even worth pursuing for, I think these people are being paid in millions
I don't care if this is a PR stunt or a highly exxaggerated contribution of claude in the entire ebola tracking scenarios, I feel very optimistic about the future around how these models are just becoming a daily life tool for everyone and not just programmers or tech bros
This actually seems awesome, just going through the video on their site made me burst out with 100 different ways in which I would tinker with it, spend time in learning different gaits, some RL or some tricks to have it jump or do a back flip
I can see some parallel between how arduino gave hobbyist a mini computer they can use in any way imaginable, I can see this kind of thing being used to solve unimaginable problems by hobbyist or RL computer scientist or heck any UG student with claude subscription
Slightly unrelated but I have no idea how wind speed predictions are made up high in the altitudes, is it just interpolation of data that are get from various radiosonoids across various timestamps and finding a regular or seasonal pattern?
Generally at altitude the wind goes from west to east. During the Cold War the US would launch spy balloons to overfly Russia whilst the wind was blowing against the Russians reciprocating. This is generally true in the current Ukraine war too.
As I understand it, it's a combination of things: aircraft reports, atmospheric motion vectors (e.g. a cloud doesn't have any propulsion, so if a cloud moves 30 km in an hour, you have learned something about the wind), Doppler wind lidar, and satellite measurements.
The numerical predictions of weather models often have many vertical components as well, so solving it for ground level also requires extending the forecast to the air, depending on the model.
To predict anything on the ground, you have to predict air temperature, humidity, wind speed and direction etc. for the entire troposphere. The output of the models in altitude are as relevant to the meteorologists as the output on the ground for manual expert analysis.
There are of course several different global weather models that predict the behavior of the atmosphere, and it is a lot more complicated than just interpolating past data. But they all do rely on radiosonde data for daily calibration of actual atmospheric conditions.
Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general.
It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
> # FRAMEWORK ARTEFACT: the run's very first transition is replayed WITHOUT advancing state
# (tools.py:954 and agent.py:468 both `continue` before `state = next_state`). So on the
# level that contains that step (level 0) our counters start exactly one action behind.
# That skipped step was action 1 with BOTH avatars moving, so seeding n=1, bumps=0 reproduces
# the framework's lagged state exactly.
# CAVEAT: this seed is only right while level 0 has never been RESET. If you ever RESET
# level 0, change the seed to n=0 (after a reset the rollout re-inits and no longer skips).
That tells me that there is some leakage between runs. The idea of ARC3 is that agents start working blind, on new tasks, via API. A RESET is counted as one action. Without seeing the actual code that produced these traces we have no way of knowing how many iterations it took, if the "framework" played the same level multiple times (comment hint above makes it likely) and so on. That's why I said that before we actually see the code / can replicate / ARC team confirms it on new envs, this should be taken with a grain of salt.
The comment more likely means the harness source was read, not memory from a previous run and the first few turns of bp35 appear to be a cold start.
Sure none of this is certain without the source.
I do believe the authors that this schema significantly improves over the base, particularly given that it took 22x simulated turns over 14 hours, which is moving the trial and error to context rather than to game. I also don’t doubt there is some contamination.
Regardless, the approach is sound and I do believe it would significantly improve scores, even if that was +20-30 over baseline (49% in this case) it does imply the benchmark is measuring the harness more than the model.
You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe.
Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them.
The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session.
