Learning the basics of how LLMs work is legitimately doable in a weekend if you have ever passed a linear algebra course.
If you are actually good at linear algebra, it's doable in an afternoon.
They are fantastically simple in structure, all the intelligence is in the massive piles of data in them, not their structure. Then there is of course a lot of room in optimization once the basics are done.
Too bad for all the horrible misfeatures that got designed in and that we still have to live with, like the entire I/O Manager/Object Manager/filters/driver stack mess, with no caching anywhere above the drivers.
IMO the next big shift in AI hardware is when some of the upcoming logic-compatible memory types mature. Inference inherently likes many small local pools of memory, with their own interface each. This is impractical with currently available memory other than sram.
I assumed the comment above is referring to operand dtype and not accumulator dtype since they mentioned ops. In general even for nvfp4 you use fp32 accumulators because you are usually reducing over large Ks which mean large values.
> I think homeowners are too close to the problem to make rational choices.
The homeowners are making rational choices. That's the problem. As the most expensive asset a typical homeowner will ever own is their home, it makes rational sense for them to support any policy that supports it's value and oppose any policy that undermines it's value. Building more housing undermines it's value.
And this is why the local community absolutely should not have any more say about a new development than a hypothetical someone who would like to live there, but can't, because there is no available housing.
> And this is why the local community absolutely should not have any more say about a new development than a hypothetical someone who would like to live there, but can't, because there is no available housing.
If you start giving "hypothetical people" a say, you stop having democracy. You can invent as many hypothetical people as you want to force any outcome.
And real democracy requires local control. A distant democratically-elected government making centralized decisions may just as well be replaced by a tyrant, given how unresponsive it will be to your concerns.
The "hypothetical people" are in actuality all the other voters in your country. Lots of people are "pro-development, just anti-development near themselves".
Do you think that the Japanese system where permitting is centralized away from local government antidemocratic?
But aren't there other salient factors? Is energy policy any less "big" than zoning in its effect on your local environment? The thing is, it's just really hard to get certain policies to work right when they are set at too small a scale -- even if they have a big effect at a small scale.
When I consider the housing situation, I am starting to think that land use policy seems to be that kind of thing.
Ultimately, cities are granted a charter by the state. It's not a given that they can control zoning, the location of people's garage doors, &c. We can think about these things and decide.
no it doesn't, high density allows more value to be extracted from the same plot of land since the volume of units offset the individual price reduction of each one. as a result, developers regularly offer homeowners 2x or 3x their market value to incentivize selling.
Yes, it more efficiently utilizes the land. But, if you are not the person who owns the plot that gets developed, more housing being built next to you harms you.
My impression is that "home value" is often a dodge that sounds better on paper than "I don't like change". You generally can't sue over hurt feelings, but if you can attach some kind of monetary value to it then the courts may take your case.
As you note home value is an extremely nebulous concept as it is tied to whatever people who currently do not live in the community might pay to buy a house. No matter what someone sells for they can always say "it would have sold for more if every single house was painted exactly the same shade of beige, you miscreants with your off white, ecru, and oatmeal houses have cost me tens of thousands of dollars!"
Density will make their home value go up. They are perhaps making rational choices if what they want is for the neighbourhood to stay the same -- but then they are accepting a lower value, by retarding economic growth.
AFAIK, there has never been a memory encryption scheme that successfully protects the VM client from the hypervisor, they've all been found to be defective within a few years on the market.
It is probably an useful layer because some of the attacks are not feasible for a remote attacker who has compromised the host but require actual physical access. But the marketing spiel of protecting your data from Amazon itself is just lies, physical access is more powerful than root access, no exceptions.
> But the marketing spiel of protecting your data from Amazon itself is just lies, physical access is more powerful than root access, no exceptions.
I'd trust Amazon and Google more than most others when it comes to stuff like formally verifying hardware security solutions and also doing active research against threats posed by malicious internal staff.
Yet less than 20% of the 80+ major vulnerabilities in confidential computing have been found by cloud companies, with the majority being found by universities and independent researchers.
It's not actually in their interest to look too closely. They know they are selling snake oil, and they do not care if it works. The only function of the technology, as far as they are concerned, is to be something they can market to customers who are scared at running on public cloud, and either they never intend to exploit it, in which case it doesn't matter if it doesn't work, or they do, in which case they don't want the vulnerabilities to be found and fixed. Either way, there is no upside to them for going spelunking.
> It's not actually in their interest to look too closely. They know they are selling snake oil, and they do not care if it works.
I can tell you from direct experience on the inside at AWS that they care tremendously about virtualization security.
It's why they never let two tenants share a physical core simultaneously; they were prepared against side-channel attacks before Spectre and Meltdown were discovered.
It's why they never supported Intel SGX, preferring instead to sequester processors for confidential workloads (Nitro Enclaves), and why modern EC2 instances support memory encryption (Graviton + AMD SEV-SNP).
It's why, when you improperly remove a chassis from its cabinet, a mechanism destroys a key component containing the machine's attestation and encryption keys, rendering it useless.
It's why they delegated a large amount of security surface, including practically all I/O functions, to their off-CPU Nitro card.
and yet they have solutions in place to work around all that to do all sorts of things with all there customers data without any customers knowledge because for example law enforcement agencies may demand that for one of there customers one day...
Fact is if you put it in the cloud you lost control. Data, services and integrity is gone. Uploading your stuff to Pirate Bay may actually be more secure because downloading it is maybe illegal :-)
> and yet they have solutions in place to work around all that to do all sorts of things with all there customers data without any customers knowledge
Such as what? Point to examples, please; don't just thread-crap hand-wavy statements like this.
Customers are free to--and are in fact encouraged to--use their own keys to encrypt their data both in transit and at rest.
> Fact is if you put it in the cloud you lost control. Data, services and integrity is gone.
The alternatives are arguably worse. You can't get anywhere near the durability of a cloud provider with any economically feasible alternative. Moreover, regardless of where you store data, law enforcement will be able to demand it be turned over with a valid warrant, and it's far easier for them to break down your door and collect it from your homelab servers (or just walk away with them under their arms) than to get into a Tier 1 cloud provider's data center and decrypt the encrypted data from it.
Either frontier AI will continue to get better steadily, until it's better than people at basically everything, and very weird things start to happen, or, it plateaus and the open weights models will eventually catch up.
There's a third possibility, which to me is also the most plausible. Models will keep getting better and better within certain domains, while never catching up with humans when it comes to other type of tasks. Like that elusive quality-without-a-name some have called "taste". Or knowing what UX is most intuitive to humans. Really, anything where we have to tap into our common humanity and our human experience in order to be of help to other humans and generally to life on earth. Which is an all-important skill, since the flowering of life on the earth is (or should be) the whole point of anything we do, after all.
We've spent a couple of millennia documenting that elusive thing called 'taste' in word, sound, image, lately in moving images. I'm pretty sure that even if you didn't have taste you could make a good simulation based on all that input.
So what you're saying is that since there are so many examples of what humans have preferred in the past for these models to be trained on, surely there will come a time when they will know even better than us what we want and what we like.
Everyone knows the path followed by every single technology mankind has ever developed - including the incredible run that was Moore's law. Sure, some will think "This time is different" - but is it?
It doesn't need to run indefinitely. I don't think anyone sane thinks it's going to run indefinitely. It just needs to run long enough. Moore's law was what, 50 years? If AI keeps improving at the rate it currently is for 20 years, the world will be very different when that curve flattens out, in a way that makes most of our current concerns are entirely irrelevant.
Those cycles have shortened considerably. Just look at the lifespan of media, integrated circuits, software, products. There have been so many big model releases just in the last three months I've lost count. We're now closer to weeks between steps.
Maybe if China stops releasing capable models the pace will slow down a little, but they have lots of incentives not to.
It is, because just like you mentioned, there's no exponential physics to back up growth. LLMs have to stop scaling up compute, because there's only so much resources.
But even if we think back to the previous decade or two (sans AI), most of tech has been a a red queen's race - developing and improving stuff, without any real substantial results.
Microservices and orchestration are not better than old monoliths. Electron apps are not better than native. Most programs people actually use have decades-old codebases.
Video games of today don't look substantially better or play different than ones from 15 years ago.
Full mask sets for leading edge nodes cost tens of millions these days, and it takes months from committing to the design to actually getting the silicon in hand.
Companies that design CPUs today commonly simulate the designs on massive server farms to find bugs before they commit to building masks. Running purely on CPU, such simulations of leading edge chips struggle to reach 1kHz, which is a problem given how long it takes to just boot an operating system these days. (And this is at full scale-out -- dedicating sufficient compute to the problem that it no longer gets faster if you use more, due to communication latencies.)
Because of this, it is now common to use FPGAs to emulate better understood parts of the circuits, to get better speeds and do more validation work in software before committing to the masks. This can get them up to a MHz.
If you are actually good at linear algebra, it's doable in an afternoon.
They are fantastically simple in structure, all the intelligence is in the massive piles of data in them, not their structure. Then there is of course a lot of room in optimization once the basics are done.
If interested, maybe start here: https://www.deeplearning.ai/courses/how-transformer-llms-wor...
reply