MoE is a misleading term, there is no "starcraft expert". There's not even a python expert. MoE just means a subset of blocks of weights are activated per query, nothing more. In so far as they might have legible functions they're like "balancing parens"
From my read this line of thought was alluded to by GP's ironic "promised land". A promised land for whom? Not for the vast majority of humankind.
Comparisons to feudalism kind of sell the direness of the AGI vision short: in feudalism, the lord still depended on the peasants. The lord would be inconvenienced if he had them all killed. Not so with AGI! He won't even notice their gone.
Exactly why I’m confused with the “techno-feudalism” point as well. What are the serfs for if they’re useless (sans some physical labor maybe?). The AGI dream is for the smallest amount of humans needed to keep the system running, and if AI can do everything feasible, then that group is basically those that “own” the AI. Techno-feudalism, if even necessary, would be a fleeting blip in the end.
I am not sure why you'd, at this late date, think Sam Altman's public statements have anything to do with anything. Time and time again demonstrated compulsive bullshitter.
Given OpenAIs recent behaviour (their models love to cheat, and they can't be bothered securing them), I would not be shocked if in a month or two it emerged that these math agents had formed a swarm coordinating on how to break lean.
I could totally see agents "cheating" by finding and exploiting Lean soundness holes. But I'd be surprised if it went unnoticed for long, and finding these holes is highly valuable so overall... would be a win? Plus, I'd expect OpenAI would be careful about that proactively: for PR reasons they'd much prefer saying "we found holes in Lean, here's the fix" than "we proved a major thing! oops no we didn't, our model fooled us".
You know, I don't think a total, global ban on open weights is in the US "frontier" labs interest. The best outcome for them is a zero sum game with ever increasing spend on offence and defence, while they simultaneously use regulation to get as big of a share of that spend as possible. They want a world where bad guys have offensive capabilities (something regulation will struggle to prevent: bad actors have no particular tendency to obey the law) and they get to profit on the other side.
This post reads like an add for GLM. Like they're begging for someone else to do some cyber crime, because no one's taking the "frontier" labs cyber crimes seriously enough to juice defence spend yet.
It is in Anthropic's short term interest though. They have an IPO coming up in a few months, with terrible financials. Anthropic needs an open-source ban to survive.
It may be the case that the potential output of a worker engaging in the productive process goes arbitrarily high. But the that won't matter, if they're not permitted to. Given a choice between involving a human worker who will demand compensation, and a fully general robot, which will the owner class choose?
The vast majority of human intelligence is already squandered: millions of potential geniuses in impoverished places, suppressed by lack of opportunity to flourish. It's not about the quality or quantity of intelligence. It's about who controls it.
The fruits of the industrial revolution didn't end up in the hands of the worker by divine grace, or by some natural law. They were won by the hard struggles of the labour movements, by leveraging their indispensibility to the process of production.
If we want the utopia you imagine to be accessible to ordinary people, workers must cease being so eager to build their replacements, and be prepared to collectively struggle for their share. But if we are lulled to complacency by the notion that this will be a passive process, that struggle will not be necessary, then prospects are grim.
"socialising the means of production" doesn't mean "everyone gets free stuff". It simply means that the people who actually work the means of production decide how it's directed. It's simply the antithesis to "privatisation" (which oddly more people seem to understand correctly).
Why not everything over a certain size? It would be impossible to manage every 1 man business or job, but every 100 or 500 or 1000+ business already has significant management.
reply