* It was Starlink, and putting it on offensive weaponry is against the T&Cs. That's not Musk's whim. That's just the rules.
* It was Starshield, in which case it was the Pentagon that throttled its access, since Starshield is controlled by the Pentagon not Elon Musk
Must we do the tribal political affiliation signalling on every HN post that vaguely involves Elon Musk?
We're left wing. Elon's right wing. We've got our knickers in a twist because an influential person dares to have a worldview different from our own. But must we broadcast this? Every time?
You're correct, the base (non-Pro, non-Max) chips are still not great for local LLMs. M5 and M6 both improved prefill speed/time-to-first-token over prior gens, but bw is still a bottleneck.
There's rumored to be no M6 Pro or M6 Max coming, i.e. the higher bandwidth chips will skip a gen. M7 is also rumored to have better memory bandwidth across the range.
Arguably the thing you’re missing is the 5090 is $1999 MSRP and more like $5000+ retail currently, and then you still need to buy the rest of the computer. This on the other hand is a whole system for $2400 (for the 32gb config). Maybe won’t set the “local LLM” world on fire, but likely to do a lot more for it than the 5090s will for the moment.
The case of your 5090 having become a coveted artifact of great rarity in these times - may be a nuance of some measure.
Also, the following two have very different 'needs' to feel good enough:
1. heavy agentic workloads via harness
vs
2. a simple private chat interface with a local instance
Even if an MBP's bandwidth were higher, it's tough to see a reason to run it as a constantly loaded 24/7 inference machine due to heat degradation.
P.S. still extremely awesome to be able to host and chat with a decently quantized 27B model on your laptop. ...and to be fair even a 32GB Air would do that!
It is a thing of beauty: the amount of folks this may add to the hacker/tinker community.
The 5090 does have a bandwidth advantage, but also must pass more data from main to GPU memory for any particular task. Comparisons of a 5090 to a DGX Spark at 273 GB/s are illustrative since the Spark uses unified memory similar to Apple's M-series. [0] The 5090 certainly wins most benchmarks but that doesn't mean the DGX Spark is not useful.
Loading data to the GPU will generally never be the bottleneck. Yes that's a bit better for unified memory, but prefill and decode will still be bound by compute and memory bandwidth respectively.
There's some value in having a huge amount of memory addressable by the GPU for loading larger models, but even that doesn't make sense for current gen models. For MOE models, newer inference engines with smarter GPU expert caching and pinned memory allow a GPU with lots of bandwidth to significantly outperform a Spark, so that leaves you with larger dense models that fit on a spark but not on a GPU where you might have a use-case.
But even then it's hard. For instance a full-fat BF16 of a ~30B parameter model will fit on a 128GB spark and not a 32GB 5090, but at that size it's going to be frustratingly slow on the spark given the compute and bandwidth constraints.
Imo these devices are mostly good for experimentation, or if you only want to do overnight jobs or training which wouldn't be possible on a GPU, and you don't want to pay $100k for a DGX station.
Is there a single topic on HN that is not subject to silly Godwins Law ad hominem attacks on Elon Musk?
Must we have to remind people that raising an arm (while saying “my heart goes out to you all”) does not make someone a Nazi, not matter how strenuously the leftwing media try to present it that way?
Just to play devil's advocate - why should a private American citizen offering a telecommunication network be compelled by an Eastern European power to let them use that network to guide killing machines?
Bearing in mind this war has absolutely nothing to do with the private American citizen.
Seems especially dangerous for the American citizen to be required to provide this support now that Russia is destroying Starlink stations in response - see the stories from Lithuania and Poland:
Just for the record, I support Ukraine in its defence against Russia's illegal and outrageous invasion. However, I don't understand by which moral or legal code, Elon Musk is required to provide direct military assistance to them?
Because there was no warning. He just crippled it and then denied that he's done it.
Ukraine uses it for defensive purposes, not offensive. Starlink is disabled on Russian territory.
Musk alongside Trump want Zelenskyy gone (receipts are there) and he thought pressuring him by crippling access to Starlink would do it or at least get him to stop attacking Putin's oil refineries.
> Because there was no warning. He just crippled it and then denied that he's done it.
What if he's telling the truth and it wasn't him that disabled it? How is it that you have obtained evidence as to internal operational procedures of Starlink/Starshield/Pentagon that make you better qualified than Musk to judge who carried out an action?
And why would Ukraine deserve any kind of warning? Surely if anyone breaks the very clear T&Cs by using Starlink in weaponry, then they should expect the service to be withdrawn, or at best, temporarily limited.
> Ukraine uses it for defensive purposes, not offensive
Ukraine have blurred the line since they started attacking energy infrastructure and civilian data centres etc deep inside Russia, with loss of Russian civilian life. I'll say it again - I support Ukraine vs Russia's illegal invasion. However, from Starlink's PoV, Ukraine is using its network for military attacks on Russian civilian infrastructure, which gives Russia the right (arguably) to attack Starlink infrastructure (e.g. ground stations). Why would Musk allow this to happen?
> Musk alongside Trump want Zelenskyy gone (receipts are there) and he thought pressuring him by crippling access to Starlink would do it or at least get him to stop attacking Putin's oil refineries.
That's a possible motivation, although speculative at this point. And it seems certain people are strenuously trying to paint Musk as the bad guy in this, and any other circumstances they can find. There's some partisan noise and speculation that we need to remove in order to find the truth here.
> What if he's telling the truth and it wasn't him that disabled it?
Because he admitted that he was behind throttling Starlink.
> And why would Ukraine deserve any kind of warning?
Because Ukrainian lives are at stake? If you owned Starlink, would you just cripple it without warning them first? This is not some SaaS service. People died because Ukraine wasn't able to defend itself from Russian jet powered drones.
>However, from Starlink's PoV, Ukraine is using its network for military attacks on Russian civilian infrastructure
Again, Ukraine doesn't use Starlink to attack Russia. It uses it to defend itself on their own territory.
>That's a possible motivation, although speculative at this point.
Trump and Musk both said Zelenskyy needs to be replaced. Trump said it on TV and Musk on Twitter. And they also parrot Russian propaganda and claiming he won't allow elections, even though it's in Ukrainian's constitution that elections can't be held when there is a war.
One of the first things I did once Claude was capable - converting all my websites from PHP to static / React/Python / React/Scala (for those that needed performance).
PHP was great for learning the basics of web development 25 years ago. The world has moved on a lot since then.
“PHP was never intended to be a new programming language; rather, it grew organically. [Its creator] Lerdorf has said: "I don't know how to stop it [...] there was never any intent to write a programming language [...] I have absolutely no idea how to write a programming language [...] I just kept adding the next logical step on the way."
I don't understand why capitalism (free markets, individual liberty, personal responsibility) should specifically frighten us in relation to AI.
Imagine the horror of the USSR, or Mao's China, or Jong Un having this technology.
Would we honestly prefer a far-left totalitarian dictatorship having control and leadership over AI?
At the end of the day, we'll always fear and distrust AI being owned by someone other than ourselves. Questions over the ownership of AI is important - in fact, too important to be conflated with our populist gripes against capitalism.
The advantage of capitalism is that more capital generate more capital. More machines you have to produce things... which will help you to produce even more machines...
The machines are operated by the labor forces, the real humans. However, the exponential compounding benefits of theses feedbacks loop us are mostly redirected to the ones owning the machines, the owner of the means of production.
Of course, the labor forces operating the machines/means of production benefits it, because they have either cheaper, or more goods. However they doesn't get this compounding effect.
AI is the god of the machines, the ultimate means of production. The amount of labor forces needed to operate it is very few. When the owner of capital invest a specific amount of money for their company, they get X amount of tokens, enabling them to work on Y amount of tasks. It's reward with low-risk.
The drift between capital owners and labor forces will become wider and wider. Until the labor forces doesn't have any more capacity of buying the goods produced. And then the system will implode.
But if goods and services become so cheap and easy to mass produce, won’t they become vastly cheaper? Thus helping to bridge the gap created by production concentration and job elimination?
> Imagine the horror of the USSR, or Mao's China, or Jong Un having this technology.
The irony of this comment, while social media and AI are being weaponized right now in elections all over the world for political alignment with the MAGA kleptocracy.
> Would we honestly prefer a far-left totalitarian dictatorship having control and leadership over AI?
As opposed to the far right totalitarian dictatorship having control and leadership over OpenAI and Anthropic?
… before the downvote, just a reminder that by sidelining Congress after the first 60 days of war in Iran, he has literally become a dictator in the original sense of the world.
And if you say he doesn’t have commandership of AI companies, I have to point out that he effectively blackballed Anthropic by labeling them as a national security supply chain risk because he didn’t like the terms of a contract
reply