It's not too late to do something about it. Acknowledging that we are not powerless to determine what our future looks like is the first step to taking control. We all seem resigned to the impending disempowerment and crisis of meaning for humanity as a whole. But this is just playing into their hands. We should collectively exercise our power of self-determination while we still have it.
How, specifically? If I derive meaning in my life, economic prosperity, and social appreciation from creating games out of my imagination and bringing them to life for people, and they can now be reverse engineered and slop-copied instantly with no credit of any sort, what am I going to do about it? Make games and never show them to anyone? Stop doing what I love? Spend the rest of my life in pointless and wasteful battles over it?
No, I'm probably just going to say fuck 'em all, I feel sorry for everyone. Go to a bar and drink while the world burns.
There's nothing any of us can do about it individually. But collectively, we still have power. It's a matter of organizing into collective action. This is a political issue just like any other. When the vast majority of people don't want something to happen, we have the power to stop it if we act. I'm not the one to make that happen; there are much better people for that. What I'm saying is how we speak about the issue should be one of agency, not of resignation. Agentic attitudes are contagious, just as defeatist attitudes are. Perhaps collective action on a small scale can spontaneously organize out of it. Or the right people might see enough people poised to act to be the organizing force. None of this happens if we admit defeat before shots are even fired.
I completely agree with some people not being able to handle free time and a good life. A lot of western countries are seeing even before AI people whining and making up imaginary pains and discomforts with a desire to burn it all down. I think the solution is death camps where people are forced to fight till death in muddy jungles, snowy tundras. I am not wholly joking, some people really need to be shown pain to appreciate life.
The nice thing about technology is that it is technology. You can try to stop some people. Countries with nukes dominate and do anything they want to countries without nukes. Go back and fuck off to the woods if you hate technology, stop bothering or dictating what others can do.
Lets start with burning down transport, people should walk, using transport is evil and removes meaning from walkers and horse drivers. Lets go all t he way back and kill the first man who invented the stick.
I don't see how. It's not an electoral matter; it's not a political system. It's not ICE coming for my neighbors, or a broken healthcare system. It's the epistemological apocalypse. It's the supernova where meaning collapses and the howling noise of madness overwhelms the signal of reason. Perhaps you could shut it down in one country, by law, but Pandora's box has been opened. The Tower of Babel. Open models will soon be good enough to create a phoney version of anything, anywhere, close enough to seeming true that by the time you discover it's not, a million more clones will be out there. Lies have always spread faster than truth, but never with such a force multiplier. The truth ain't dying in darkness, it's dying in blasting white noise, diffusion, ironic that turned out to be the approach. Or maybe not. What's this land rush to claim latent space but the attempt to expropriate, forever, the human ability to discover it?
The Leonard Cohen line keeps coming back to me: The blizzard, the blizzard of the world has crossed the threshold and it's overturned the order of the soul.
Sorry, but... I scoff at the idea that collective action can do a damn thing about it. There isn't even a collective idea of what to do instead of it. Collective action also has heavy connotations of communism, under which, people like me were also not permitted to write games and reap the economic and social benefits of doing what we enjoy. But at least one could say that, for the communist bloc, the collective action finally came against communism. So there's that, and maybe there's some sort of small-creative-capitalist-producer collective action that could be taken here...
but who are we kidding! I just had dinner with a friend who's a VP at [major motion picture studio] who described the situation in the movie business as "apocalyptic", and they have the ability to organize and a history of organizing far more than whichever random cross-section of white collar knowledge workers and engineers loses their livelihood in the next six months, or the six months after that.
No, my friend, I think this is a war where no shots needed to be fired, because we were all so dazzled by the bright lights that no one realized we were being steamrolled until it was too late. It's already over.
When you need a football field of GPU to host frontier models and advance the state of the art, it's not too late. The supply of GPUs can be monitored and controlled. Access can be limited. International agreements to stop further progress can be hashed out (the China bogeyman is a mirage). We could outlaw the commercialization of more capable models and seize datacenters that aren't in compliance. If the current models were as far as we got, we'd be mostly fine. The only thing stopping us from realizing this future is the inability to coordinate.
So much intense whining to the point of performing terror acts just because a technology can work some things better than you. Why don't you rather split humanity into anti-ai and indifferent to ai and let us see who progresses to where in a few decades?
Progress fetishism will be the end of us all. There is a point where human prosperity is not further served by more "progress". I don't claim to know where that point is. But simply treating progress as some universal good leads to disaster. But it is clear to me that human disempowerment is bad for everyone.
My worry isn't about being made obsolete, it's about human disempowerment. Of course you'll just say AI is a tool and so forth. My point is about where things are going, not where they are currently. AI boosters are rarely willing to extrapolate progress into the future in its defense. It's either willful ignorance or strategic feigned ignorance.
I think ai is potentially conscious already but thats neither here nor there. I think we can keep giving challenges to people. Space is a good one. But eventually there has to be some solution to weaker minded people. I think experiencing real pain for a set amount of time is probably the only real solution. Because we are already seeing what too much comfort does in some Western societies even before AI. Another example would be people who jumped a financial rung in life or achieved some lifelong goal. We already have examples.
>and the end user/society at large captures the lions share of the value as real work done.
This is a fallacy where you extrapolate the large change in a single dimension and imagine everything else staying mostly the same, ignoring the disruption of the system from non-linear effects of that single dimension being saturated. When a few companies capture a large proportion of the economic value of the system, the economy simply doesn't exist as it is currently conceived.
The current economy is driven by the need for broad participation to maintain the collective standard of living. We all get to skim a bit of that value for ourselves, and so distribution is inherent to the system. When broad labor participation is no longer needed to create collective wealth, the leverage of the average person to demand a share of the value produced evaporates. The economy in this scenario is something very different. There's no reason to imagine wealth being distributed in an egalitarian way; it certainly won't be automatic.
The ideal human society is not one where people are narrowly focused on creating widgets or increasing efficiency, narrowly construed. An ideal human society includes intellectual pursuits. Some of these pursuits do not suit dabblers, they take deep expertise that can only happen from a lifetime of directed effort. We are collectively better off when some value is spent in ways that isn't aimed at maximum efficient allocation.
It's funny, the justification for accelerationism is that people can be paid (UBI) to pursue their passions. On the other hand, we have those that are asking why anyone should be paid for work that isn't economically viable. I suspect we're being sold a load of bullshit.
I don't support a UBI, for what its worth. The idea is nonsensic from my vantage point, if there's a certain minimum level of good people should have access to it makes more sense to give it to them rather than to give them some level of cash that can on theory buy it.
More importantly, I'm curious how you define an ideal human society. How do you land on what ideal is, or what it would take to get there?
>More importantly, I'm curious how you define an ideal human society. How do you land on what ideal is, or what it would take to get there?
The question is in what context does the human psyche thrive? We find meaning in personal struggle and overcoming obstacles. We're happiest when we're striving for something out of reach but seemingly attainable. We're much less happy when we have all of our needs met but have no source of meaning or fulfilling struggle. Our evolutionary history is one of struggling against hardship and finding personal meaning in the process. This is a core feature of our psyche and it is to our detriment to eliminate it.
This isn't to say we should all be struggling for our next meal. Much like cats, our core drives can be exercised and satiated through a simulacrum. There are many ways it can be satisfied, but strive we must, against something with inherent meaning to us. The problem with AI fully realized is that it takes away most avenues for human achievement, thus the struggle loses all meaning.
>However, I do believe that when a couple problems get solved that massively benefit all of humanity — e.g. a massive cancer breakthrough, etc — a tipping point will be reached and public opinion will reverse and enthusiastically embrace AI, and will accept not only diminished personal meaning but outright technocratic rule.
Is solving cancer worth ceding human meaning and human autonomy to the machines? I haven't seen anyone ask this question but it seems the central issue of this revolution. Personally, I don't think it is.
Perhaps machines will lead to a perfectly homogenous society completely absent of negative valence whatsoever. But if every experience is also devoid of any personal meaning, what's the point? The optimal human state is not one of peak hedonism, or wholly absent of suffering.
> Is solving cancer worth ceding human meaning and human autonomy to the machines? I haven't seen anyone ask this question but it seems the central issue of this revolution. Personally, I don't think it is.
Solving cancer alone? Perhaps not.. Unless you have cancer.. Solving all diseases and aging? Well... I would rather be alive and live like an AI's housecat than dead and free. And I suspect that many people who publicly disagree are just performative, and would actually make the same choice when push comes to shove, and they have to actually choose between "diminishing their autonomy to the machines" and imminent death.
The Christian influence on the world has been undeniably positive. Seeing death and suffering as the enemy and organizing society around minimizing it has been a boon to human progress. But now we come to the dark side of this view. A fear of death so complete that we are willing to sacrifice everything of value to prevent it. Our society has become so divorced from our natural state that death is seen less as an inevitable part of life, and more as a corporeal enemy that must be resisted at all costs. As it turns out, too much of a good thing, whatever it is, leads to absurdities and atrocities.
No, I do not think human autonomy should be sacrificed to defeat death. I personally do not fear death. It's as irrational to fear as it is to fear a deep sleep. I do fear dying painfully. Tragically, we as a society resist alleviating the pain associated with the process of dying (by speeding it up) for the same reasons. If only we could learn to take our principles in moderate doses.
> death and suffering as the enemy and organizing society around minimizing it
> A fear of death so complete that we are willing to sacrifice everything of value to prevent it.
> a corporeal enemy that must be resisted
I am afraid that you know nothing about Christianity. The beliefs and values you describe are antithetical to the Christian life and our worldview. If you've ever just examined the life of St. Teresa of Calcutta, you'd understand how absurd it would be for her to "fear death" or "minimize suffering".
We seek to follow, and imitate, Christ Jesus in his life of suffering, his death and resurrection. We do not resist, or minimize, or fear any of this. To do so, would be quite heretical to us as Christians.
Of all times of the year to bring this up. All Saints Day, celebrating the eternal life of those, named and unnamed, who have preceded us in death. All Souls Day, to pray for all the departed souls who suffer in Purgatory, after death but before their Heavenly reward.
I don't know what religion you are trying to describe, but to me it sounds like atheistic secularism. It is atheism that fears death, and utilitarianism that seeks to minimize suffering. Not Christians. Goodness, gracious.
To understand my point one must separate the metaphysics from the practical effects of the belief system. Christianity represented a step-change towards humanism. This must be considered in the in the era it was founded and compared to the competing ideologies at the time, i.e. paganism and its various death cults. Christianity institutionalized ideals like equality of men, the sanctity of life, charity, self sacrifice, etc. These ideals are the precursors to civilization.
The particular ideology of Christianity was super effective at laying the groundwork for modern civilization. Of course, these ideals have since evolved and have since become some kind of secular ideology. Granted, it was imprecise to attribute a fear of death to Christianity. But the sanctity of life is in practice a close cousin of a fear of death. Modern secularism having no metaphysical alternative has made this fear explicit.
> I suspect that many people who publicly disagree are just performative, and would actually make the same choice when push comes to shove, and they would have to actually choose between "diminishing their autonomy to the machines" and imminent death.
Only those that aren't in a field and see its existence justified for purely instrumental reasons can think its value is merely in its practical output. Personal meaning and fulfillment come in many forms. For mathematicians, part of that value is through solving problems, but it's also partly about the struggle itself, gaining understanding, expanding the frontier of human achievement, being a part of a community that values these things, playing a valuable role in the progress of humankind and making a career out of it. Having all that swept away overnight will lead to an existential crisis for anyone who is on the wrong end of this process.
It's quite bizarre how many people feel so personally affronted at the existence of a rather exclusive intellectual community that they take glee in seeing it decimated overnight. But make no mistake, AI is coming for the things you value as well.
Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior. This is in contrast to polysemanticity where reading the state of any number of neurons may not give insight into what state the neural network is in and controlling the network by activating/suppressing neurons is nearly impossible to do reliably.
> Neurons having narrow semantics means you can inspect what a neural network is "thinking" with confidence
Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
> and even constrain its space of behaviors by activating/suppressing a small set of neurons that have very straightforward effects on its behavior.
You can influence behaviour by doing this kind of thing, yes, but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
>Not really, because of emergent behaviours and also because there are still millions of neurons being summed (with variable weighting depending on inputs) at each layer.
Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist. In other words, there are no "hidden thoughts" in an LLM encoded in higher order combinations of features. Inasmuch as non-semantic features of the input influence output, it does so in non-semantic ways that are degenerate in terms of the quality of output. The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output. But the relevance of this kind of degeneracy to LLM output is at the margins. Tracking activated state predicts subsequent output to a high degree of accuracy.
>but it isn't straightforward at all, and is generally very messy and you get all sorts of unpleasant sides effects (e.g. abliterated models not performing as well).
I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
> Emergent behaviors in terms of seemingly random combinations of features producing unpredictable but coherent/intentional output basically don't exist.
You'll need to elaborate here, because Navier-Stokes plus Mathocalypse reactions are extremely strong evidence for coherent emergent behavior coming out of these models.
> The only thing emergent in LLMs is degenerate state leading to a breakdown in tracking relevant features of the input and a reduction in quality of output.
This is so provably wrong with two seconds of Googling / AI-asking that you must be using some completely idiosyncratic definition of "emergent" here.
> I expect this has as much to do with our lack of precision in our tools rather than with the concept in principle.
Unfortunately in this case it is the exact opposite.
>you must be using some completely idiosyncratic definition of "emergent" here.
I doubt my usage is idiosyncratic. Rather, the term is so ill defined that people often use it inappropriately. Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts. My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons. This isn't something I can prove, but it certainly isn't obviously false from just considering recent mathematical results. Higher order/non-linear interactions are characteristic of this kind of inexplicable behavior from complex interactions we characterize as emergent. But there are good reasons to think this kind of non-linearity in any semantic regime is next to impossible in principle.
Semantic geometric representations are characterized by linearity in response dynamics such that a linear change in one variable results in a linear semantic change in related variables. That's how vector embeddings capture semantic relationships. Call it the linearity hypothesis. Non-linear dynamics in the embedding space just destroys semantic relationships in most cases. There is no way to keep an entire network of relationships sufficiently correlated if non-linear response dynamics is a part of the representational toolbox. Non-linear dynamics in general kills correlations.
Testing this against recent LLM proofs, what would need to be shown is that the relevant neural activations do not decompose the semantic content of the proofs (or the reasoning trace) in a straightforward manner, but rather the reasoning trace is unpredictable from the semantic content of the activated neurons that precede the tokens of the reasoning trace. But this idea strains credulity; it is almost inconceivable that this would be the case. We should be highly incredulous of the idea that complex non-linear interactions can be wholly responsible for the long-running coherent output in LLMs. The default hypothesis should be that there is no emergent behavior (as I defined it) in neural networks.
The way I interpret emergent is 'not something that was obvious from the properties of the parts', so you could study the parts in isolation and never realize that there was this much more complex behavior that would only emerge if you placed those objects together beyond a certain scale.
> Emergent means something along the lines of not easily predictable behavior from the behavior of the component parts.
Yes, partly. But it also refers to complexity arising from the component parts that requires additional tools (formulas, descriptions, etc) to meaningfully describe. Or at least, emergentism has always formally involved more than mere predictability from parts (https://plato.stanford.edu/entries/properties-emergent/). I am talking about weak emergentism here, so no need to get into metaphysics and mysticism.
> My claim is nothing in LLMs is emergent in the sense of being inexplicable from an ideal consideration of simple/linear aggregations of its neurons.
IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc., but human beings are not very predictable in many cases, and also exhibit consistent behaviors that tend to be very hard to predict (or even explain) from basic knowledge of the brain, drives, and body. LLMs are no different, and what they are doing today couldn't be more clearly emergent, even by your definition.
Also, something doesn't cease to be emergent even if you can perfectly explain it with reference to the parts. E.g. attractors are emergent behaviour, even though they fall out of very simple formulas and so in a sense are completely due to their defining parts. But describing the shapes of e.g. the Mandelbrot set or the Lorentz Attractor requires more than the base formula that generated them. So they are classic examples of emergent phenomena. Likewise, a person's intelligence might one day be fully predictable from biological and genetic knowledge and measurements, but it wouldn't mean that intelligence stops being an emergent property that is still observable and useful to use itself in other models.
But your argument here I cannot follow clearly, because LLM behavior is clearly emergent in any sense of the term, including yours. EDIT: Or basically, LLMs show weak emergence, but if you disagree with this, then it would seem to me you perhaps disagree with the concept of emergence at all.
>IMO this contradicts even your minimalist definition. Almost no one predicted simple LLMs would be where they were today, and you can't currently predict their behavior by looking at their weights.
Whether some trait is emergent shouldn't depend on the state of our knowledge. That no one predicted how far LLMs would go is a demerit for our model of them and doesn't imply they have extremely complex "emergent" interactions. My conception of emergence is asking whether something can be "easily" predicted from an ideal understanding of the component parts. Simple/linear aggregations of the correct semantics of precursor neurons is what is at stake here. In other words, is understanding the semantics of the precursor neurons the vast majority of the hard part in understanding what comes next? Past failures of prediction do not bear on this question much at all.
An example from the article that is informative to my view is the association the network made to the base64 encoding of plain text substrings. If you had asked me, I would have assumed the network decodes base64 live as it traverses the layers. Instead (at least some of) these associations are precomputed during pretraining and recorded in the form of shared affinity in neurons. Active computation and precomputed associations are on two ends of the intelligence spectrum. The human brain is strongly biased towards precomputed associations. It seems that LLMs are as well. The more I see about their inner workings, the more I suspect they operate mostly in the form of these explicit associations. The computations they actively manifest across the network can't be that complicated due to depth constraints. There just aren't enough compute steps[1] for the kinds of complex, non-linear, unpredictable dynamics you get from compute heavy regimes.
>Something doesn't cease to become emergent if you can roughly explain it by making references to its parts. We can broadly explain how human behaviours result from drives + sensory processing + associations and etc.
When I talk about explaining some phenomena, I mean it in a robust sense that e.g. gives you the capacity to predict downstream behavior with high accuracy.
[1] These discussions always seem to have difficulty distinguish the computation that realizes the system from the computation the system as such engages in. There is a huge amount of compute that realizes the feed forward pass of an LLM. But the computational steps directed by the LLM to resolve its output from the prompt is severely restricted by the layers in the forward pass of the network (or however many virtual layers from looping).
Synthesizing complex information from a range of textbooks enough to build bioweapons is cognitively impenetrable for the vast majority of people. Having the AI synthesize this information into a step-by-step recipe to follow is within the capabilities of most motivated individuals.
I'm always suspicious of people who claim expertise/special knowledge but aren't quick to offer it and instead engage in petty back-and-forths. Instead of trying to win this debate in the narrow sense, why not just make the best argument you can in support of your position? If authority has any value, it is because it gives you specialized knowledge that lets you judge the likelihood of speculative scenarios better than laymen. If you can't communicate that knowledge and the argument that leads to your conclusion, your supposed expertise just isn't worth much in this context.
The raw materials to make bioweapons are easily purchased on line with no pre-emptive tracing. It's absurd to discount the expertise factor in being a bottleneck, which LLMs have evaporated.
> It's absurd to discount the expertise factor in being a bottleneck, which LLMs have evaporated.
Mmm. Especially in this arena, LLM assistance is like The Anarchist's Cookbook. A quarter of the time following the instructions will seriously injure you... and if you know enough to identify which instructions are the hazardous ones, you know enough to not need the assistance.
> But it's not a limitation that will remain indefinitely.
The Sun is going to fail in somewhere between many hundreds of millions and a few billion years. This will either turn the surface of the earth into slag, freeze it, or both. Either way, all life on the planet is doomed. This fact is not a reason to fail to switch from hydrocarbon-burning electricity generators to photovoltaic, fission, wind, hydroelectric, and geothermal electricity generators. Extinction events that will happen in the extremely distant future shouldn't prevent us from doing the things that are smart to do in the medium- and long-term.
But, -to bring things to the present day- companies that are solidly on track to hit their promised growth targets don't come out and publicly say "We're working on WMDs. [0] We are incapable of safely working on these WMDs. We refuse to stop working on these WMDs. However, if you lawmakers make special laws and regulations just for us and include us in the process, we'll be quite happy to submit the stop work order to our employees!". That's a statement you only make if there's no way in hell you're going to keep your promises and you're willing to risk jail time and annihilation of your companies for a shot at being able to con Congress into giving you an ironclad excuse to fail to keep your promises.
Given enough time and focused effort, we will end up with widely-available automated librarians that are very good. We're not there yet, and -based on current events- are absolutely not going to get there in the near future.
[0] Anything with a 10% chance of destroying all humanity is a WMD.
It's strange how quick people are willing to dismiss concerns with horrible reasoning as long as it suits their biases. Perhaps a little intellectual honesty is called for considering the stakes? There are many plausible reasons why we haven't seen AI fuel bio-terror attacks yet. None of which implies it's unlikely or implausible in principle.
But hasn't bio attacks already been possible before AI, what has changed? Maybe the barrier to entry lowered, but one could argue the availability of info has not been the bottleneck for as long as the internet has existed.
The threat is ofc real, but AI won't magically "do the thing" still, it's not code that's the bottleneck AFAIK? Correct me where I'm wrong.
Synthesizing complex information from a range of technical sources enough to build bioweapons is cognitively impenetrable for the vast majority of people. The ability of AI to synthesize this information into a step-by-step recipe to follow is a step change in accessibility.
There's an opportunity cost to terrorism just like with any time sink. The effort to build up knowledge enough to produce some weaponized pathogen will be compared to just doing traditional terrorism. For some low capability terror cell, its easy to see how the cost/benefit analysis has been in favor of traditional terrorism up to now. The kinds of terror acts that take years of sustained effort to execute are rare. But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.
> Synthesizing complex information from a range of technical sources enough to build bioweapons is cognitively impenetrable for the vast majority of people. The ability of AI to synthesize this information into a step-by-step recipe to follow is a step change in accessibility.
I agree with this part.
> But as the barriers to entry to bioterrorism fall away and become widely accessible we may see the cost/benefit shift.
But I argue that the barrier to entry is not about information, it's about needing to build, execute and maintain a lab and production facility? It's not the theoretical part that's the bottleneck.
Or maybe I'm overestimating the difficulty of implementing such a thing, this is not in my area of expertise.
>Or maybe I'm overestimating the difficulty of implementing such a thing, this is not in my area of expertise.
Not mine either. But there is a vocal contingent of experts in biotech that have been sounding the alarm for the last couple of decades at the potential for rogue groups to develop weaponized pathogens for terrorism. And this was years before LLMs. Apparently constructing a sufficiently capable lab isn't really a bottleneck due to how freely available equipment is by mail order and how unsophisticated the processes are in practice.
>Consider our publication earlier this year, in which we reported that 36 of 38 companies shipped pieces of the 1918 influenza genome. That was collectively enough to generate the virus using reverse genetics several times over – and they were ordered by a student using a pseudonym falsely claiming to work at an office address. No one asked whether we had permission to work with 1918 influenza (a select agent), or why we needed genes from it, even though the official U.S. guidance at the time instructed them to do so. In fairness to the companies, that guidance was voluntary, and the fragments we ordered were legal to produce and sell domestically in every nation of the world (export controls are complicated). That is still true today, which is why we have a problem.
reply