I strongly recommend avoiding HN for intelligent takes on AI internals. This place is full of people who think they can understand these models by vague analogies to the brain and their own pet (and almost-always underdeveloped and philosophically ignorant) theory of mind, and also that you can have a meaningful understanding of these models without at least knowing some very basic linear algebra.
I would recommend going on Twitter (or a Nitter clone) and looking at e.g. Neel Nanda, Nick Camarata, roon, Jon Stokes, or other researchers that are directly involved in this stuff. HN when AI is involved is full of the most incompetent and embarrassing armchair speculators saying things so incoherent they aren't even wrong https://en.wikipedia.org/wiki/Not_even_wrong.
Here is a measured take from Neel Nanda, who is quite publicly involved in serious interpretability research:
> This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted!
> This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!
> The proposed SoC was extensively validated on two intracranial electroencephalography (iEEG) datasets related to psychiatric disorders, achieving 80.9%/74.8% sensitivity and 74.2%/73.1% specificity on the multi-source interference task (MSIT) and the emotional conflict resolution (ECR) task, respectively. We further evaluated the SoC on the public DEAP dataset for electroencephalography (EEG)-based emotion recognition, achieving 70.5%/72.5% sensitivity and 69.8%/66.3% specificity for the arousal and valence states, respectively. [https://ieeexplore.ieee.org/document/11685197]
I dunno, really need to see the comparative performance of other classifiers and the base rates in these datasets to know whether these performance metrics are anything we should care about, and whether e.g. a classifier on simple verbal probes to the subjects might simply yield better predictions than going through the EEG signals. You can do some really interesting EEG biofeedback stuff, so maybe this is useful, but I would think the full EEG head-gear would also provide much richer and more useful information than a measurement based on a single chip. Presumably this would be for psychiatric disorders or brain injuries where the core dysfunction we know is limited to a very specific region(s).
> LLMs are showing us that a lot of what we think of as cognition is actually part of the elaborate symbol system which is Language
This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.
Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.
Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since language is only a single, limited aspect of real cognition, this makes them so annoying and alien to read.
I think we're in violent agreement here. Language is part of how we think. Definitely not the whole thing -- we are physically and biologically instantiated and LLMs are not. Language is a software, but the hardware does matter.
But LLMs DO show us that language is a big deal. It isn't just a serialization protocol for real thought -- it's a material component of how we think.
In many ways I think LLMs are the strongest possible proof of structuralist and post-structuralist theory which argues that human cognition is conditioned on the language and concepts to which we have access.
The emotion / cognition distinction is somewhat poor and often breaks down, because you can't really have one without the other (different mental states just involve differing degrees of both, phenomenologically). They aren't really causally separable, but are only really phenemonologically separable (cognition and emotion have different feels / textures, but almost always co-occur).
Most importantly, you rarely have "cognition-free" emotional states, because emotions always involve some mental model or representation of the self / world / self and world, i.e. they involve a cognitive state. Only in very extreme cases (drugs, perhaps some cases of persistent depression, etc) might you be able to have emotions that are largely uncorrelated with any kind of particular cognition.
Likewise, the same is almost certainly true for cognitive states, in that we tend to think about / model things because we need to evaluate those things. Evaluation too seems to rely heavily on emotional processes, in that damage to emotional processing tends to make rational / logical decision-making severely impaired (e.g. Damasio's work in the 90s, or https://pmc.ncbi.nlm.nih.gov/articles/PMC2396787/). Regardless, you can't remove evaluation / valuation from most thinking, and that is fundamentally linked to emotion.
Doesn't mean you couldn't potentially build a machine / thing that detects mental states, mind you. But it likely won't be very useful if it is just using simple folk-psychological terminology and concepts (or using such concepts will likely not result in much insights into the machine outputs).
I was taught it was "BOW-ba" or "BOH-ba", but AI actually linked a number of studies that seems to confirm it is supposed to be and usually is "BOO-ba" (i.e. bulging eyes Pepe) in experiments. I don't personally think either pronunciation would change the results though.
Either similar underlying processing machinery (due to auditory and visual processing evolving from simpler and more interconnected processing circuits) or because when thinking of frequencies of waveforms in light/sound, visual sharpness and auditory sharpness really are the same thing (so even if they didn't co-evolve, it can be more efficient or useful to have some single underlying representation waveform representation that represents these things similarly).
> The main difference between the two experiments was that dogs in study 1 showed a general bias toward looking at the spiky shapes. That bias disappeared in study 2, when their owners were wearing headphones.
... so, they accounted for it, and the control group (group wearing headphones) showed no significance in the dog's behavior. Seems like a huge claim when their control group to rule out the hans effect failed.
It isn't trivially obvious, no, but the "k" sound is a fricative (EDIT: oops, nope, plosive - see response to this comment), i.e. the tongue rapidly moves close to the palate and then away, creating turbulence, and sounds with a sharp peak and amplitude. I.e. the waveform of a "k" or "kiki" sound looks more spiky. The opposite is the case for the "bouba" sound.
It is a bit mysterious why animals would all map e.g. sharp auditory waveforms to sharper / spiky lines (kiki figure), and then smoother sounds to smoother lineforms (bouba figure), but, reasonably, these senses evolved together and used far more shared underlying processing machinery than they do presently. After all, both light and sound reach us in waves and both carry energy.
Some primitive mush-blob perhaps just started processing all incoming energy waves together, differentiating e.g. smoother vs. more staccatto waves before making the distinction between e.g. sound vs. light. Or I dunno, but IMO it makes more sense when you consider evolution, and don't just look at processing now in isolation from the likely development of these senses.
> The 'b' sound is a plosive too, by the way. It's voiced, vs the voiceless 'k'.
Yeah this I know, by "bouba" being opposite I meant the waveform.
Really the phonetic terms here are not actually relevant and have betrayed me. It's more that the 'k' sound is "more [ex]plosive" in that it seems to build up more pressure and turbulence (probably as you say, because the consonant is in the rear, so pressure builds and bounces around in the mouth more, whereas the "b" just mostly blasts out the front more quickly). And the "k" surely has a more peaky / spikey waveform in the audible range.
If you, or anyone else reading, is interested in playing around with this stuff, there's an open source program called Praat[0]. I assume it still is the go-to in linguistics for making spectrograms, it was back when I was studying it, at least.
There's probably nicer programs that can do the same thing, aimed at people like musicians, but they won't have the phonetics-specific guides and help online.
There's also a large archive of speech recordings from a wide range of languages hosted by UCLA[1], which you can download and play around with. There might be newer archives now, but I remember this one, because Peter Ladefoged wrote the proverbial book on introductory phonetics.
Linguistics generally aligns quite pro-copyleft, so there will be a lot of freely available stuff online.
k is a plosive, not fricative: there's a discontinuity in the flow of air, not a friction-sounding continuous airflow. that fits your point though, regardless.
Oops, you are right! Thank you for correcting. I checked before posting that some "k" sounds are indeed fricatives, but they definitely aren't in "kiki".
I would recommend going on Twitter (or a Nitter clone) and looking at e.g. Neel Nanda, Nick Camarata, roon, Jon Stokes, or other researchers that are directly involved in this stuff. HN when AI is involved is full of the most incompetent and embarrassing armchair speculators saying things so incoherent they aren't even wrong https://en.wikipedia.org/wiki/Not_even_wrong.
reply