That kind of certainty vs. understanding is limited to mathematics, of course we’re almost entirely certain about many things out in the world we don’t fully understand.
And even in mathematics, incomprehensible but valid proofs have been a thing since the four-color theorem decades ago.
Riemann hypothesis is "the statement". The entirety of it, a single statement. And we know for a fact that single statement - the entirety of Riemann hypothesis - is true. Understanding how to prove this one statement is understanding how to prove the entirety of that one statement. And like you said yourself, no human understands the whole of this one statement. This literally never happened before.
What about the classification of the simple finite groups?
That's also one statement that no human understand the entire proof of. (You can make new statements by taking multiple existing statements and combining them with 'and' for example.)
This hypothesis was previously not proven at all. That's strictly worse.
Now it is understood by our systems.
Soon it will be by us. Our systems gave us a new and working salient.
We're building machines that help us understand faster.
There have been plenty of times in history we didn't completely understand something yet used partial knowledge to make advances. This isn't even that bad.
The inverse of what I said might be easier to understand. Up until this week, there has been no statement in all of science that we simultaneously knew for certain that it's true and that nobody understood how to prove. It was always either-or. Either someone understands how to prove it, or we didn't know for certain.
> I get the concern, but isn't that a bit melodramatic?
No. This is a real risk and we should talk about it.
I got banned from Anthropic for unknown reasons a few months ago and then mysteriously reinstated. I still don't know what happened, but I was locked out for days. I had to ask for a manual review to regain access.
This is NOT okay.
The core capabilities of the future should not be owned and controlled by singular private entities. These should be open weights that we can run on whatever provider we elect.
We cheapen the concept of fundamental human rights if we start slapping the claim on everything.
Inference is a technology and using it is a privilege not a right.
That doesn't mean its a good thing that the tech is consolidated in the hands of very few. There are plenty of issues with the industry that may be dangerous if we can't fix them, but not because this tech that is only a handful of years old is somehow now a god given right we all must have access to.
imo, You are technically correct but practically wrong. Yes, we should scope things but scope creep here is harmless and the upside is concrete. Worst case it will prove to be economically impractical in which case we have data point on whether societies should be re-engineered around AI everything.
There are always potential downsides. If we convince people that inference is a human right, for example, but then collectively find its also impractical to offer or enforce, it opens the door for misreading that as human rights are impractical to defend at scale.
Will really GenAI be able to do everything intellectual "better"? I personally do not think so, but we'll need to wait and see.
OTOH, as far as access to "inference", your brain inference is still unrestricted. Even today, some industrialized products have their very expensive "hand made" variants — at least that niche remains. And that's just the worst case where AI does beat human intelligence at everything.
While tool access is important, currently the bigger restriction is compute than weights — and this has always limited what type of work and research you can do. Eg. how many supercomputers are there in sub-Saharan Africa and what work was limited with fewer computers?
On a new credit card, with a burner phone… and that's before they really went through with personal identification as was required by short lived export controls in the summer.
The reason for no aliens could be that the universe is fragile, and once you find the magic of its construction, you unlock vacuum collapse, FTL paradoxes, and construct black holes.
Smart aliens just kill themselves because the structure of the universe is a cliff.
> 'most of the code in faang today is written by AI'.
And it's true.
We're getting to a point where you don't even have to look at the code anymore, and I'm serious.
> AI had not gotten to a point of where it could outperform humans but in this experiment they find out a group of agents are good enough for this task
The more this is done, the better it gets. Especially as more resources are poured into making it better.
It's only a matter of time before AI auditing is the common practice and humans are the backup for the AI.
This. I literally do not care about any of their offerings unless they are open weights.
I'll pay Microsoft to run them if they're open weights. I don't want to deal with the hassle of doing that myself. But I won't even touch them if I can't possibly own the weights myself.
The market is already full of closed weights. We don't need more.
Can i have more explanation please? I'm not in risk management but i would like to understand the situation better. Just risk that they change it under you? Or that they're training on your data, or what kinds of risks are we most worried about?
They can make your model disappear. If you have a system and a set of prompts and evals and everything is working well, it's no fun for that model to just disappear. Some will reply "do a better job building your system so it's not model dependent" and they are sort of right, but that isn't always economically feasible and/or just isn't the reality of how a lot of systems get built.
Models hosted as a service can be subtly tweaked without any announcement, or suddenly be 'upgraded' to a new one with a ticking clock on how long the old one sticks around. In both cases, the flows people establish around them are disrupted or broken.
I've lost track of the anguished cries of those unhappy when either of these has happened.
That's a problem with decision models, for example, but coding models have no such risk. I switch between multiple coding models from multiple providers each day.
This assumes material switching costs. I use frontier max plans for now but have fail over models for each class of model action with tuned context so I can keep the performance when I lose the max plan subsidy at a lower price that staying with the frontier labs for certain classes of work.
I see minimal downside in using closed weight models as long as they're behind a router, you have good prod sourced evals and you have fail over to other models configured and tested (running either dark or active:active) with real prod workloads.
If you want relatively repeatable and predictable responses (even within the reality of being non-deterministic) then making sure you'll have continued access to a model to replay responses for debugging and tuning can be important.
reply