It was 6 hours where the opposing counsel tried to rattle me / get under my skin with insulting questions.
To be clear, I was only an expert witness in the lawsuit (drug + pregnant lady + birth defects). It paid extremely well, but was something I would never want to do routinely.
At the end of the deposition, the lawyer had to catch a flight to the west coast and she asked ME if I would drive her to the airport* (she said she'd pay me my hourly rate)!
At that point it was clear that the insults were just theater and "part of the job", but I took it all too personally and told her I had to get home.
* She was late for the flight and Uber wasn't a thing yet.
it's not that they don't take it personally and actually feel the pain/ disgust / fear / , it's that they hold the tenets of law as absolute and the most important part of the system. This definitely takes a particular type of person.
This is because Christianity played a major role in Western law. While the laws and courts predates Christianity, after the fall of Romans, much of it was rediscovered through Church.
Many of the courtroom customs were partly influenced by this. For example the Robes/gowns or the emphasis on "law as an absolute", higher standard that sits above personal feelings is also inspired from Christian natural law ideas (i.e rulers and judges under a higher law) even though those ideas were built on Roman foundations.
So the kind of person who can treat the rules as more important than their own feelings isn't that surprising in that tradition. It requires a certain bit of.. religiosity.
whenever some unsavory person is in the news being defended by his attorney, most of us get upset, but they say stuff like, "his attorney is just doing a good job defending his client. that's how this works."
Second time in as many days that I’ve seen someone use that quote without any apparent understanding of the topic at hand nor what they’re copy-pasting.
I recall something I once heard about product marketing. The experiment was a product like jam, and they had 3 flavour options available in one case and 12 options available in another. It was a test on how hard it was to make a decision as well as how confident the person was they made the right decision later on.
One of Sartre's points on existentialism was that radical freedom leads to anxiety.
In some sense, AI is removing scarcity. It isn't clear to me that this is the same kind of thing as going from starvation to surplus. It is more like the jam case where we aren't increasing abundance in a total sense, we increasing the options within the existing surplus.
There are a lot of things that people are capable of doing themselves that they pay other people to do for them.
I agree that LLMs have and will continue to increase the number of activities that people can DIY. But I believe many people will still choose to pay for others to do things on their behalf.
I don't think it is self-serving. Tao has stated plainly that he thought LLMs would go the route of assistants, like co-pilots. But what he feels now is that they are actually being engineered to be replacements. His claim is that the way OpenAI dropped these math results isn't "assisting" anyone, so we should all read between the lines.
Consider any man at his job, fighting the good fight. Your boss says "I've hired this man to assist you" and you are happy to finally have some help with your difficult task. Then one morning you come into work and see that the coming day’s work was done overnight by this other fellow. And the realization sinks in, he isn't there to help you, he is there to replace you.
In this view his reaction seems totally expected. I expect many professionals, e.g. doctors, lawyers, accountants, etc. will make this realization sooner or later. The companies are saying these tools will assist professionals, but in reality they are aiming to replace them.
> The companies are saying these tools will assist professionals, but in reality they are aiming to replace them.
If anyone doubts this, consider that Anthropic considers its TAM to exceed $30 trillion which is nearly the entire GDP of the US and 25% of the world GDP.
Only makes sense if they believe they will massively replace labor. Which, to their credit, they used to admit openly and publicly and only stopped because some crisis PR people told them "hey maybe you shouldn't say the quiet part out loud while the hoi polloi still have the power to stop this from happening".
Do their own hiring practices reflect that ambition? They sure seem to be recruiting a lot of humans for typical roles in business, legal, and engineering. If they want to flex the power of their models to automate human jobs, why not lead by example?
> They sure seem to be recruiting a lot of humans for typical roles in business, legal, and engineering.
They have practically infinite investor money to throw at salaries (lost in the noise of how much they are spending on compute) so why not use it for now to reach their goal as fast as possible?
My thesis isn't that they have already achieved the ability to replace massive amounts of the labor pool, just that that has to be their goal given their stated TAM.
> If they want to flex the power of their models to automate human jobs, why not lead by example?
They don't want to flex this. They realized (after a false start) that it is better for them to avoid talking about this in public.
That doesn't mean their goals changed, they just realized that flexing before the goal is achieved could endanger the goal being achieved through political backlash.
>They have practically infinite investor money to throw at salaries (lost in the noise of how much they are spending on compute) so why not use it for now to reach their goal as fast as possible?
Wouldn't throwing that money into GPU be better because AI is smarter and faster ? So they would reach their goal even faster ?
I’m not making a claim one way or the other about the intent of these companies, but if your product made labor 3x more productive, you could still easily justify a TAM of the entirety of US GDP, without resulting in anyone losing their jobs. Obviously, here in reality, AI does act as a near 1:1 replacement for people in some roles, so we should expect some job losses. But I don’t think there’s any reason we should expect the entire US labor force to be automated away any time soon, even taking these assessments at face value
You do realise Anthropic's management, after throwing its tantrums, gets a seeing to and has to do what its investors want? Which is pump up the valuation.
Isnt OpenAI and specifically Altman openly talking about that being the goal?
> The companies are saying these tools will assist professionals, but in reality they are aiming to replace them.
They are openly saying that the goal is to replace humans and make humans obsolete. Like, literally openly. I am not saying I agree with that goal or find it realistic, but they do think so.
A few-minute search on Google shows that Altman has said a lot of contradictory things on the subject. A few quotes, as examples:
“AI won't replace humans. But humans who use AI will replace those who don't.”
vs
"Some areas in the job market will be 'just like totally, totally gone' as they're replaced by AI agents.”
and
“AI will probably replace most of the jobs people do today,”
The most recent position I could find says he has revised his earlier position on a total replacement due to the lagging adoption of AI in many industries. But I don't think there is a single position we can point to that they are "openly saying". It is fair to grant even the CEOs of these companies a change in position as they gain more information about capabilities and adoption, rather than insist that they hold, have held and will continue to hold fixed opinions.
The point is that Tao's understanding has changed. He may have been naive before, he may have heard in their words something he wanted to believe. Perhaps he was misled by people who he trusted that worked in the company. But now that he has more evidence, in the form of the maths drops and the way they have been handled, his suspension of disbelief no longer holds. The companies could have done this in a way that included mathematicians, so they chose not to. And that choice reveals their aims unambiguously in a way that a media-trained CEO or careful PR department can avoid.
That suggests that Tao is not being "self-serving" as others claim. He is saying that his optimism in the best-case scenario turned out to be incorrect.
The reason we fell first is because we embraced the copilots. We used them heavily, this gave them more and more training data on top of the code they slurped up from our using their assistants. Soon the assistant became the agent who killed the human.
They also had a rich starting point from open source code (which we somehow gave up without realising it). All of this won't happen as efficiently for these other fields, but it will happen, much in the way you've described. We (swes) turned out to be a rather dumb group of people, which is hilarious since we think we're clever.
The only moat we have is the human to human element, we need to talk to other humans in the business. Customers, etc. But once they're replaced by machines or automation, then machine to machine communication will be the preferred mode of communication and work and the legacy humans will be increasingly cut out of the loop. Human in the loop is a death sentence we just don't realise it.
Tenured at age 24, I doubt he has any experience in industry to understand what he was getting into. Which makes me even more incredulous as to why people worship him.
You might be mistaking my comment as a value judgement rather than an attempt to charitably explain the latest media communications from Tao. I detect in his position that some trust he granted the AI companies has been violated.
You are, of course, right that doctors, lawyers and accountants have more control over the environment they work in compared to mathematicians. But it doesn't change the underlying argument Tao is making. Your observation that those professions will react differently is likely to be true and only time will tell how effective their response will be.
But AI "assistants" is a marketing approach being made to all professionals at this time. Tao is implying this is a Trojan-horse strategy. If you let the assistants in without reasonable protections, you may lose control over the environment you are trying to protect.
you may not be aware of it but society largely runs on the results of maths, clueless indeed. and regulated or not, if there are financial savings to be made anyone can be shoved under the bus.
This may not be helpful or strictly accurate, but when I see deontology or variants of that word, I substitute "SHOULD". It is like a duty or obligation.
NOTE: This would suggest a surface reading of Brandom's "deontic scorekeeping" might be keeping score of who has what obligations, who is following or breaking rules, etc.
Then the argument against LLMs being party to that is that we don't really assign roles and responsibilities to LLMs, they don't really have "duties" or "obligations" in the way a member of society will have. We don't negotiate with them over who should do what, they don't contribute to the rules that are set for them.
I'm literally just guessing based on the context here though, since the few times I've tried to watch podcasts with Brandom I generally bounce after 15-20 minutes feeling I haven't really understood anything he is saying.
Perhaps you can also help me understand why #1 being right and #2 being wrong would be some win for alignment that people on X claim it to be?
My impression, being a total outsider, is that the reaction feels like a bunch of insider bros hyping up their insider bro friend. It feels more like PR than a genuine groundswell behind a truly revolutionary idea.
The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."
How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.
Because if each neuron represents a small set of similar concepts instead of several completely different things, it should be easier than expected to look inside the neural net and see what it's really thinking.
It might still be quite difficult since the network is huge, and just because you know what the individual neurons represent doesn't mean you understand the complicated thoughts they're building. But easier than expected is still good news.
If (2) is correct then you need to train SAEs and other things to figure out what a neural net is thinking. This is very expensive and slow, which is why a lot of neuron-interpretability experiments are done on small networks. The techniques don't scale.
But if (1) is correct, and Nick presents strong arguments that it is, then you can toss out all the indirect stuff like training dedicated interpretation networks. So the techniques can scale and be trustworthy.
Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior. This is in contrast to polysemanticity where reading the state of any number of neurons may not give insight into what state the neural network is in and controlling the network by activating/suppressing neurons is nearly impossible to do reliably.
> Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
>Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
> Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
>you must be using some completely idiosyncratic definition of "emergent" here.
I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.
Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.
Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.
The way I interpret emergent is 'not something that was obvious from the properties of the parts', so you could study the parts in isolation and never realize that there was this much more complex behavior that would only emerge if you placed those objects together beyond a certain scale.
> Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts.
Yes, partly. But it also refers to complexity arising from the component parts that requires additional tools (formulas, descriptions, etc) to meaningfully describe. Or at least, emergentism has always formally involved more than mere predictability from parts (https://plato.stanford.edu/entries/properties-emergent/). I am talking about weak emergentism here, so no need to get into metaphysics and mysticism.
> My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons.
IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc., but human beings are not very predictable in many cases, and also exhibit consistent behaviors that tend to be very hard to predict (or even explain) from basic knowledge of the brain, drives, and body. LLMs are no different, and what they are doing today couldn't be more clearly emergent, even by your definition.
Also, something doesn't cease to be emergent even if you can perfectly explain it with reference to the parts. E.g. attractors are emergent behaviour, even though they fall out of very simple formulas and so in a sense are completely due to their defining parts. But describing the shapes of e.g. the Mandelbrot set or the Lorentz Attractor requires more than the base formula that generated them. So they are classic examples of emergent phenomena. Likewise, a person's intelligence might one day be fully predictable from biological and genetic knowledge and measurements, but it wouldn't mean that intelligence stops being an emergent property that is still observable and useful to use itself in other models.
But your argument here I cannot follow clearly, because LLM behavior is clearly emergent in any sense of the term, including yours. EDIT: Or basically, LLMs show weak emergence, but if you disagree with this, then it would seem to me you perhaps disagree with the concept of emergence at all.
>IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Whether some trait is emergent shouldn't depend on the state of our knowledge. That no one predicted how far LLMs would go is a demerit for our model of them and doesn't imply they have extremely complex "emergent" interactions. My conception of emergence is asking whether something can be "easily" predicted from an ideal understanding of the component parts. Simple/linear aggregations of the correct semantics of precursor neurons is what is at stake here. In other words, is understanding the semantics of the precursor neurons the vast majority of the hard part in understanding what comes next? Past failures of prediction do not bear on this question much at all.
An example from the article that is informative to my view is the association the network made to the base64 encoding of plain text substrings. If you had asked me, I would have assumed the network decodes base64 live as it traverses the layers. Instead (at least some of) these associations are precomputed during pretraining and recorded in the form of shared affinity in neurons. Active computation and precomputed associations are on two ends of the intelligence spectrum. The human brain is strongly biased towards precomputed associations. It seems that LLMs are as well. The more I see about their inner workings, the more I suspect they operate mostly in the form of these explicit associations. The computations they actively manifest across the network can't be that complicated due to depth constraints. There just aren't enough compute steps[1] for the kinds of complex, non-linear, unpredictable dynamics you get from compute heavy regimes.
>Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc.
When I talk about explaining some phenomena, I mean it in a robust sense that e.g. gives you the capacity to predict downstream behavior with high accuracy.
[1] These discussions always seem to have difficulty distinguish the computation that realizes the system from the computation the system as such engages in. There is a huge amount of compute that realizes the feed forward pass of an LLM. But the computational steps directed by the LLM to resolve its output from the prompt is severely restricted by the layers in the forward pass of the network (or however many virtual layers from looping).
I do wish I could give it a try with some free credits but I would bet Google is wanting to limit usage until they know they can handle the traffic.
I'm starting to feel the subscription fatigue and a small amount of FOMO. I just can't justify pro accounts for every frontier model now that it includes OpenAI, Anthropic, Meta and Google.
I enjoyed this talk very much, although I suspect that the capability work in Scala preceded this new agental AI world and just happens to apply, rather than being formed by it. Like a solution that found its problem.
But his approach seems very pragmatic, not judging whether or not agental development practices are good or bad, but rather considering how we can address the issues they create at the level that he works at: language design. No doubt his work in practically realising these ideas will contribute to the evolution of language and systems design.
The benchmarks I saw posted to X are impressive, which makes them hard to believe.
I recall that Meta made massive investments into compute infrastructure that put their capacity near the top of all competitors. If all of this is true, it looks like ownership of compute is a significant differentiator in performance.
Yeah, especially non-coding benchmarks which for Meta might be more important long term than coding ones as they are going to deploy it to billions of non-programmers.
1. https://www.youtube.com/watch?v=C7UC2ie0XIE
2. https://www.youtube.com/watch?v=g1j4N-3ZkB4
reply