There were obviously some internal politics going on—Google joined the Alliance for Open Media, a consortium of the who's who of the media/tech industry: Apple, Facebook, Microsoft, Amazon, Netflix, Nvidia… the list goes on. They supported AVIF as the next generation image format.
Google apparently wanted to be one of the "cool kids" supporting AVIF. When they removed the JPEG XL code from Chrome, they created the excuse that there was little demand for it, along with some other disingenuous talking points that didn't quite make sense. Ironically the project that ended up becoming JPEG XL was started by Google employees in Switzerland.
Anyway, I’m glad Google is supporting JPEG XL now. I wanted to use JPEG XL files for a project I started a month ago, but it was a non-starter with global usage under 20% at the time.
Removed because having yet another giant C codec was a huge risk, re-added because the PDF Association chose JXL as their next image format so anyone rendering PDFs (including browsers) is now required to have it.
"Required" is a strong word. More than enough people use Chrome as a PDF viewer that if Google decided to not support JXL, you'd see ~0 PDFs with it.
Another giant codec is only a huge risk if you run the decoder in the renderer process, which they say in the linked thread they do, but I don't see a sane reason why.
I'm sure if they asked the secret Gemini 4.0 AGI to use their existing very good sandbox to throw all the decoders in their own process and only share the image buffers, it could do it in a few hours.
They also love keeping malware apps on the play store as well. I tried reporting an app that took over one of my parents phone. It stayed up for quite a while. I'm sure it's still up.
"App store experience" was continually degrading. I used to be able to find small apps and games I liked a lot and basically binge downloading. The last few times I tried, I ended up with nothing useable.
Which massively slows down the output. Doing this with Qwen 9B already takes you into seconds per answer territory, and Jev is supposedly frontier level intelligence.
They do work, but transformers work better on the accuracy/compute tradeoff curve. Transformers let far away positions in the sequence exchange info in 1 step, while a dilated CNN will need many layers. Additionally, Transformers can be fairly efficiently trained.
I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.
It isn't so much that "other setups didn't work". More like "the first thing we found that did work turned out to be the minimal thing that could work".
Transformers are conventional feed-forward neural networks alternated with attention blocks. You can think of them as big huge "ordinary" neural networks augmented with this new kind of block.
Attention blocks are basically just a differentiable hashmap. Think of it like a scratchpad memory.
It turns out that a hashmap/scratchpad is pretty essential to being able to untangle language. I don't find this too hard to believe. Somewhere in there, you have to build the graph of which object is acting via which verb on which object.
What is surprising is that this is all it takes! These simple little hashmap/scratchpad units (and massive scale) are really the only thing you need to tack on to a feed-forward neural network to get essentially general intelligence. This is totally surprising to me.
If I had to summarize this in with sentences then here is my attempt.
Transformer training can be parallelized easily, making it possible to use brute force to train the neural network quickly (bitter lesson rewards compute friendly scalable architectures).
Transformers have perfect retrieval, they re-read the entire context window from scratch for every token.
Explanation over.
If you extrapolate this, then the logical conclusion is that the next model architecture would use even more brute force.
Right now transformers can only append a token at the end. This means they can read any input, but write only one specific output.
If you wanted to extend this, you would want to make the transformer read from any input and write to any output, i.e make it capable of updating the entire KV cache every iteration.
most people in ML have no idea what transformers actually are.
Traditional networks, at every layer, used to be output = [weights matrix][input], where input is a vector, and weights matrix is the weights, where each row corresponds to the set of weights for each neuron.
Transformers upscale the dimension of the data. Instead of the above, transformers do [output] = [input][weights_matrix]. When you multiply an input by a matrix, you get an output matrix back. Thats all that happens. Nothing fancy. You have weights matricies for K/Q/V, which when post multiplied with the input, give you the KQV vectors, and then you just simply multiply them together and apply a scaling factor.
There is nothing magical about K/Q/V. There is nothing about any one doing any querying or any one representing some keys. The naming is just a carry over from how they that selection process is used in pre llm data science fields where you manually define the key and query matricies to define relationships between components.
The reason of why it works is because is an extension of something called kernel tricks from pre LLM machine learning days - you map a lower dimensional space to an extra dimension based on some equation, and it lets you apply some classifier on the combination of existing values and new value. Thats what transformers are doing - they are mapping the individual token to the dk x n_heads latent space, which allows for a higher dimensional representation of the data, capturing complex relationships.
You can do Transformers with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on. The thing is, there really isn't any way to tell if any of that gives you more advantage - it certainly would give you more granularity, but as of right now, in terms of training to generate a specific token given previous ones before it, it seems that you don't need any more dimentions than dk x n_heads. Interestingly enough, you also can mathematically represent any such transformer including the starting one with a sequence of linear layers like in traditional networks, the only thing is that it becomes computationally inefficient due to having duplicates of data.
The reason why RNNs and others and others didn't work is because RNN training is effectively trying to linearly regress on chaotic effects - i.e what set of starting conditions would evolve with a given process into what you want. This is an NP hard problem, and you can't really do it linearly.
Transformer models on the other hand, use breadth instead of compute to capture interactions. In those learned weight matrices, you have a latent space of a bunch of "knowledge" compressed, and an algorithm to search on that "knowledge".
But, its very possible that an RNN can be smarter than a frontier model while being much smaller in size - in the same way that its very possible that you can have the right set of prompts for an existing local inference smaller model that can basically be very close to AGI in terms of being able to solve any problem across any domain. Right now, the space is about exploring those prompts, which is the frameworks and harnesses, to get to there, as well as making the compute portion more efficient so you can explore that space faster.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
Hey I like your exposition of attention in terms of the kernel trick, but the big huge difference is that kernel methods use the inner product which is a commutative operation -- it's bidirectional (and both tokens are projected into the space by the same function prior to being dot-producted).
This means that it can't capture unidirectional relationships, like "ball" is the object on which the verb "threw" acts in the sentence "I threw the ball". This relationship is true in only one direction; it isn't true to say "threw" is the object on which the verb "ball" acts.
I do think it would be fair to say that transformers generalize the kernel trick to noncommutative relations by applying a different projection function (W^Q and W^K) to the two tokens being considered. This makes the overall operation (project then dot product) a noncommutative operation.
And the thing that comes after harnesses/efficiency in terms of progress should be obvious if you understand all of the above.
Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!
I mean, given sentence construction, you don't really need to capture directionality, you just have a mapping of how sentences are constructed to the latent space of some representation.
>Nirvana? Singularity? Paperclips? Vernor Vinge rising from the dead? I'm curious; please share!
Simulated evolution. Thats how you "solve" highly nonlinear chaotic systems. And generally, if you think about it, you have to have some secondary system on top of the knowledge embedded in LLMs to drive them to select certain tokens, which then starts to eerily resemble what humans call emotions in themselves.
I mean, given sentence construction, you don't really need to capture directionality, you just have a mapping of how sentences are constructed to the latent space of some representation.
Uh, no, you absolutely do need directionality.
Without directionality you're simply grouping words into equivalence classes. That's nice, but graphs are strictly more powerful than equivalence classes.
Simulated evolution. Thats how you "solve" highly nonlinear chaotic systems.
Eh. I dunno. Evolution is a horribly inefficient way to do learning. Nature uses evolution because it's the only learning algorithm that you can implement using uncoordinated chemical reactions on DNA/RNA/proteins.
As soon as evolution hits a point where it can build neurons all the effort switches over to gradient descent. Just look at humans.
Even cellular/self-organizing systems can be driven more efficiently by gradient learning than evolution... if you want your mind blown, read the paper that this page summarizes:
Im saying you don't need to capture directionality if you capture all possible cases of sentence construction.
As for evolution, you can still go gradients, the problem is that you can't do gradients in a space with many false positives. You need some method of figuring out the true optimal point.
> You can do this with 5 matricies instead of 3, you can do this with 4-dimentional tensors, and so on
As a outsider I have many dumb quesion like these. I am trying to understand transformers in a Occam's razor way. It's a complicated machinary after all.
Imagine you have a soccer field, a ball with position x and y, a kick strength, and direction in an angle. Your job is to write a function that determines if the ball will end up in a goal. So that is 4 values. However the function itself will contain many intricacies, like trig functions, simulated drag, and so on.
In the contest of LLMs, you cant have these types of coded function. Your function has to be a mathematical equation that is smooth - i.e no discrete steps, no singularities. The reason for this is when any neural net is trained, you use backpropagation of the error to adjust weights, and how much you adjust them is directly proportional to the weights effect on the final output, and in order to compute this, you have to have smooth functions from start to finish.
So what you do instead is you add data to your 4 values, that capture different relationship between them. If your 4 values are x,y,k,and h, your first data point can be a1x + b1y + c1k + d1h. The second point can be a2x + b3y + c4k + d5h. And so on.
You can have as many of those values as you want. And then you can add, combine, and scale those values in any way you chose.
This basically gives you a map of 4 values into a binary decision whether the ball will end up in a goal or not, after sufficient training. However, the total number of extra values that you chose has to be large enough to capture all possibilities - if you don't have enough, you will start to make mistakes for some initial conditions.
Very interesting analogy, thank you! When I initially learned about linear regression, i learned that to capture non-linearities, instead of choosing a more complex, non-linear hypothesis function, I can just come up with "arbitrary" features for my data set. Basic example: house price calculation. Obvious features are square_meters, age, n_rooms, ... But I can make even this linear model learn complex connections by transforming or combining these input features and add them as additional inputs, such as n_rooms * age, or log(square_meters) or whatever.
What you're explaining sounds very similar. Is it, or am I understanding it wrong? (Idk why it's so hard for me to understand this attention thing...)
The ancient wisdom which advocated for stored procedures, which modern developers find distasteful, were really advocating for microservices close to your data, which encapsulated security, business logic, and data persistence so that multiple consumers could share the same data without repeating the logic and code.
The fact that some people write those microservices in PL/SQL and some in JavaScript doesn't change the relevance of the encapsulation.
But PL/SQL is a horrifying language. You have barely any facilities for modularity, encapsulation or composition. Every procedure is just one name in a huge flat name space. Development tools are primitive. The language is absurdly verbose - it was state of the art in the 1970s when COBOL reigned supreme.
That's the theory. I never ran across a project that used stored procedures which didn't have a code base split between stored procedures and whatever server language, with no rhyme or reason for what goes where. It's just a maintenance headache.
Removed because no one from Google benifits from supporting it then.
Re-added because someone could get a promotion by supporting it now.
reply