In the twin cities, one big brewery, Bauhaus, went out of business briefly and got bought and reopened by ABV, a company that specializes in NA, alcohol-adjusted, and THC beverages, https://www.bauhausbrewlabs.com/blog/Blog%20Post%20Title%20O... . Now you can get beer in full strength, session, NA, and THC. It seems like a sign of the times, but I'm happy to have more choices and as I get older I don't feel like drinking the strong IPAs like I used to.
yes, that was surprising to me too. It would be a big deal if they switched to an encoder-decoder model like the original transformer. But I don't think that's what it's doing. One thing is the causal part, so in the original transformer, the encoder was bidirectional, but in this case it is not, so that's one difference. So I think it's an optimization for the prompt/prefill so that the attention is summarized into the output of the encoding layers, rather than all the layers. I just skimmed the paper too so if anyone else has insight, please correct me.
I think that given the developments in AI in the last ~10 years, it would be hard to say that attention is not part of consciousness. I'm not sure if it is "all you need", but I think that any theory that doesn't include it would be very doubtful.
I think the parent is referring to agents as outward exploration that may come up empty-handed, but I see LLMs and agents as inward exploration, trying to define what is attention, knowledge, intelligence, consciousness, agency, etc. So in this case it is the terra incognita of the human "soul" that LLMs are exploring.
