I agree Opus 5.5 is something different entirely. It is even hard to find tasks for other agents to be honest where you know the results will be picked apart when Opus 5.5 reviews them. In my setup Opus 5.5 uses codex from time to time when I'm running out of credits and it decided to run 2 astra agents for the same task to improve its performance.
> I agree Opus 5.5 is something different entirely.
If model versions are anything to go by, I'd have assumed Opus 5.5 and 5 were built on the same pretrained core, but the vastly superior performance of 5.5 makes me wonder if Anthropic literally pretrained a new one using Fable (following their RSI advances).
Opus 5.5 may be the first RSI'd model and it shows.
It does it through codex CLI. It is basically a tool that accepts a prompt. There is a way for claude to chat with codex as a person would do but thats slightly complex than a CLI tool.
I don’t buy arguments like much faster than X. Take X let claude work on it for 2 days and it will be fast. In case of gpui I took the community version as a part of my project and it is quite fast because it is optimized for my app.
At some point comparing human chess to algorithms stopped making any sense. So there is no realistic pool of games between AI and humans. We would lose every game and vs AI a human ELO would go to 0 (not sure if that is technically possible). It is like this for probably 15 years.
ELO is like probability. A 100 ELO difference is something like 2x odds of winning a game and over a lot of tries it tends to be quite accurate. That was also an argument from Magnus Carlsen that to compare human ELO you need a lot of games and he didn't want to take part in classical world championship quoting this reason (among others).
I agree. It is hard to make a separation between what is possible thanks to programming vs hardware. It is only easy to analyze post factum that certain ideas worked. Maybe there are also other factors that are important - time, availability of hardware, knowledge dissemination. What would be possible if the hardware stopped improving.
At this point model upgrades do not mean too much for an established use case. I have 36 benchmark scenarios using agents + tools in my app and the results were 30/36 for gpt 6 luna, and 33/36 for gpt 5.6 luna. The benchmark was tuned for gpt 5.6 luna but still apart from slightly reduced cost I will keep the default model to gpt 5.6.
What I'm missing here is also type guarantees. I don't think you can do it without token level logic which forces the model to output the tokens from a predefined pool of tokens. A logic like this given some JSON schema is not that difficult to implement. If the LLM must output JSON schema compatible value then you can also add that it doesn't "hallucinate". Which is funny too because just guaranteeing the type does not mean the model does not hallucinate but this is another story.
It is changing so often that to keep the costs and quality at a pareto level you need to experiment with a lot of different providers. And the pain you are describing should be a normal process in AI. I use model pools where the tasks are optimized for multiple AI providers. For example I was experimenting with GPT Luna and it turned out that the model is good but tool shy so I had to improve the instructions. Now this model is my main model for the chat in my app. Next week it can be a different model.
I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.
Any compressor actually can be used, trivially, as an autoregressive language model.
Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate.
You can easily generate text with gzip this way. It won't be very good text, because gzip compression is not as sophisticated as a transformer + SGD, but the principle is the same.
The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.
This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.
For example, LLM pretraining datasets are on the order of tens or hundreds of terabytes. If an LLM-based code for that data is ~twice as efficient as gzip, you could afford to transmit the weights of even a very large LLM and still come out ahead.
In other words: LLMs actually are excellent compressors of their training sets in the formal information-theoretic sense.
>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
You can, actually! Any compressor can be losslessly converted into a generator, and vice versa.
Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.
I agree that technically they are doing the same thing but in practice LLMs are better compressors than PNGs (learned this while I was researching this topic). That was quite surprising to me.
That sounds like boostrapping the weights involved in early layers, to obviate the need for those layers to learn (optimize) for the distribution in the training set.
Makes me wonder idly,
- is this conceptually akin in some sense to a "universal grammar," and if so
- with a broad enough training set, is there a latent durable universal grammar that might be similarly recovered and injected to the benefit of all training,
- does that grammar go beyond morphological/syntactical/grammatical features, into e.g. semantics and pragmatics
That was my thinking as well mainly to increase the speed of training. But it may turn out that the simple statistics that you can capture like this may account for 1% of the training and are likely to be captured as the first thing during the training.
But actually these techniques are used but they are hidden as speculative decoding with increasing complexity of approximations. For example you can have a part of the network that predicts the next word based on the markov chain, the next approximation is more complex etc.
reply