Musk and Zuck both have far more voting power in their companies than Jensen Huang does in Nvidia. Zuck specifically has effectively unilateral control.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
The thing I had to wrap my head around is: For a "classic" classifier, the classes are fixed and the data distribution is p(x,y). The calibration is checked over this single distribution. Jev makes the tasks part of the input, so the data distribution is p(task,x,y) and calibration is averaged over that joint distribution. What follows is that while for classic classifiers, calibration brakes down if your x was out-of-distribution, it can now also break if your task is out of distribution. The big question now is: Can subtle differences in phrasing your task already lead to this? And the only way to verify calibration is having ground truth data to validate on.
Also: Calibration over p(task, x, y) means calibrated on average across tasks. Even an in-distribution task can be over- or underconfident while the average looks fine.
> And the only way to verify calibration is having ground truth data to validate on
I’m not sure that’s true in real cases.
I would expect that in distribution tasks have quite consistent outcomes with variations on phrasing. I would expect that to be much less true when tasks are out of distribution. So could we randomly adjust phrasing and see how much the results change - looking less for truth but consistency?
A model can be consistently over- or under-confident for similar inputs. The only way to check if it's actually calibrated is to compare against some known ground truth. I don't see any way around that.
In my experience, calibration is only ever in expectation, even for a classic classifier. Any given prediction is often still over- or under-confident.
> It makes an unbounded number of future predictions
nit: it's trained on finitely many samples.
> So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
Huh? That's not counter to anything I said. All I said was it's not trivially implied.
Wouldn't learning a calibrated probability distribution just be a consequence of the CE loss being a proper scoring rule? So, by construction, they are approximating the distribution of the labels.
Of course in practice there might be issues with overfitting the log-likelihood and the with the labels being a finite sample of the population distribution
Yes, in a sense, with some important caveats. Cross-entropy will tend to yield reasonable marginal (i.e., not class-wise, not approximately conditional) calibration for in-distribution (i.i.d. data). The challenge with LMs, and neural networks more generally, is that the resulting estimators that are based on the output logits are very brittle to covariate shifts and out-of-distribution data, and that for real applications, we often need robust estimates in high-probability regions (e.g., of class- and prediction-conditional accuracy at least some value near 1) for conditional-branching decisions (by a human or a model). (ECE is the wrong metric for that quantity.)
The various decision APIs don't give you that, and would themselves need to be recalibrated with constraints in feature-representation space over the applicable reference class, as with https://pypi.org/project/reexpress-sdm/.
> Well marriage is really mostly a religious thing so you can’t say it’s not about religion.
That's a non-argument against a social construct. Religion has influenced every aspect of society, including morality as a whole. That historical fact doesn't invalidate the social norms, in the same way that secular norms are not invalidated by their dissociation from religion.
> Imagine being almost 60 and worrying about your wife finding you had an affair. I think you should rather worry about the next episode which is certain death.
Because of his actions, he will be alienated from his family. Objectively, pretty much every human derives fulfillment in life, and peace as they approach death, from their relations to others around them.
> Give him a break.
LOL. Let's give this poor billionaire a break, think of what these HN comments will do to his psyche! What's that about his wife being cheated on? Shush.
> Controversial take: AI music is more interesting than the autotuned, quantized, mass-market-optimized slop that dominates contemporary pop "music".
> And it was genuinely novel in a way that pop music isn't. I will sacrifice a lot of pride for novelty.
For one thing, "interesting" and "novel" are not particularly the point of art. An infinite TikTok feed of AI slop is in some sense infinitely interesting and novel.
But also, those concepts are subjective. Pop music has a ton of evolution and innovation if you break it down. Sure, not every hit is doing something clever, but plenty of them are. You might not find those features to be interesting but it takes ingenuity to stand out in an era where the barrier to entry is so low.
> I suspect that for most people, much of it is a feeling of accomplishment from performing a difficult skill that creates something of value for other people. It's the joy of being known as a good cook, and the satisfaction of making something for people that they might otherwise have trouble getting. How much fun would cooking be if every street corner had a Michelin-starred restaurant giving food away for free? How much would you cook if none of your friends really cared about the results because they can get better food elsewhere, cheaper?
I think it's paradoxical to suggest Michelin-starred restaurants could be everywhere, given the whole point is the travel-worthiness i.e. a scarcity that justified effort.
That aside, to take the strongest form of the argument, I think even if every human being could get their ideal meal, no matter what it is, at any time and for free - and even if the cook in question is only doing it for social reasons - cooking would still be valued. It's the effort put in by a person which is scarce and precious. Even if people were immortal, that time in the present would still be valuable in itself.
> How is kindness demonstrated in a world where no one wants for anything?
Post-scarcity of material things won't sedate humans. Kindness is a thing wanted in and of itself, but more importantly, internal struggles and desires can't all be resolved through material gain.
> How can someone be more interesting than synthetically generated content created specifically attuned to our personal interests?
The answer is apparently, pretty easily? By and large it seems the consensus is social media and digital content are exploiting a niche of providing a low-effort, highly-addictive pastime.
...no? Whatever value they got from inference they already did is still there? Of course the government still needs to worry about vendor lock-in, but they'd have to do that no matter what, and also general-purpose LLMs are relatively replaceable for most use cases.
Given the statements in their SEC filing, it's not outlandish to consider the possibility of a long-term plan to make Anthropic models a nationalized, public utility before they go bankrupt.
If GSA is already considering vendor lock-in as they establish new public services across multiple agencies, the natural move is to build them to eventually transition to a federally-owned and operated provider. Especially with how involved the U.S. government is in limiting capabilities Anthropic provides to consumers today.
If the company has a sustainable business model it will be profitable and hang around without jacking up prices. If not, you wasted time and money setting up the anthropic tool chain.
Except that work would be cheaper to set up and the cost would drop over time if they were using local llms. Every dollar paid to a 3rd party rental service is a dollar that is burned in the fireplace. We are paying more for the right to be stuck paying more forever, it's ridiculous.
Reviewers are also usually biased towards reviewing products that are good and/or interesting, and while that's also true of consumers, you'd expect reviewers to be more informed.
What you're describing is a language that can effectively take a high-level design doc (limited to a formal language representing a pseudo-English-subset) and compile to almost arbitrarily complex, formally verifiable systems. I say systems because you're also implying these could essentially orchestrate distributed systems.
Which, if that were really possible, would be pretty good evidence of AGI. Or, at least, the end of 6-7 figure software engineer salaries. Conversely, I think it's a pipe dream. Writing a compiler that actually works well across such a broad range of hugely complex behaviors would itself be absurdly complex.
AGI will accelerate it, but way before agentic development I learned this was the future when I worked at Amazon. I no longer have access to the internal talks about it (amazing to watch) but here is a public link worth reading:
You don't really need compilers anymore. LLM's can write programs over any AST and provide an equivalence proof provided you have good semantics for the source and target.
Yeah, but thats a lower level of abstraction, thus using more tokens, and more importantly occupying a larger chunk of the biggest current limitation, the context
Semantics of a language describe the behaviour of a sentence in that language when executed. When you have semantics in a theorem prover you can construct equivalence proofs between programs expressed in different languages (or proofs that a compiler from one language to another is semantics preserving).
I was mostly referring to accuracy when I said "good" above, in the sense that they should reflect the actual behaviour of the language they describe.
Check Inform7 for text adevntures. No AGI nor AI needed. On Inform6, it's a dumb simple OOP language where the structure it's almost self-evident. Classes, in-game objects with real-life attributes (enterable, a container for other objects, non-movable/pickable scenery and so on).
PD: inform7 can run on an RaspPi B+ or 25 year old PC's like nothing, and i6 down to 30 year old ones. The ZMachine adventures themselves, from any 16 bit computer and up.
Also you would probably want to cache the compiler's output in git because running that compiler would take a long time, and you probably want the compiler's output to still be inspectable to human inspection. In the end that compiled artifact would look a lot like source code.
I think Rust's success in the AI space shows how useful formal proofs are, as Rust provides something very similar to that. Both because it helps with the understanding of LLMs, as well as with trusting the LLMs are not violating any basic properties (like please no nasal demons if the user sends a buffer larger than the allowed amount).
```
SELECT A, B, Sum(C)
FROM Somewhere
GROUP BY ALL
```
Very few developers look at the underlying “source code” that this SQL compiles into. Very few people even read the “source code” that tsc outputs.
The reality is the lowest level checkpoint worth looking at and understanding is the highest possible abstraction layer which preserves 100% of the needed precision and flexibility that the product being built requires.
Also, you’d just use something like sscache or just .o files to speed up compilation. It’d be unusual to store compiler MIR/LIR in git.
I think the important part will be figuring out what we need from the higher level language that achieves 100% of the precision and flexibility of Rust for the product being built.
> Very few developers look at the underlying “source code” that this SQL compiles into. Very few people even read the “source code” that tsc outputs.
Only because there’s a group of people that do understand database theory and know how to write a proper implementation and they have a track records of doing so. Themselves relies on compilers maintainers and OS developers, and this goes on down to the semiconductors where people do study how electricity interact with the materials.
Every layers of the tower of abstractions have people that understand the current layers, knows the safe semantics of the layer below, and can provide a safe foundation for the layer above.
> formal proofs are, as Rust provides something very similar to that
Since when? Rust just has an ML-like type system, and seemingly many have met that for the first time with Rust, even though it has decadeS of history, and multiple production languages running it just fine (ocaml, scala, etc).
Formal proofs are way way above an ML type system. Like how do you encode mathematical statements in Rust's type system? What about statements about program semantics, like program A is equivalent to program B <- in formal verification THAt is the type of an expression, and the implementation of it is the proof.
By similar i meant the strong and powerful type system that Rust has, plus its safety guarantees (iterator invalidation is a compiler error etc). That is in the end a proof system: you are proving to the compiler that you really don't mutate the buffer and you can't drop the T in &T, etc.
Of course the Rust language doesn't give you tools to prove that the program is bug-free or such. The proofs only extend to a basic set of properties, but there is a proof system in Rust.
As for other ML languages, they aren't really C/C++ competitors performance wise, Rust is.
We've got inform7 with an English like language generating complex inform6 OOP optimized code since a few decades and the i7 compiler might even run under a Pentium II with 128MB of RAM under NetBSD. I6, under 32 and a Pentium. And maybe even less. ZMachine games themselves (the biggest v8 ones) under the Frotz interpreter, under a 486 and 8MB (with FreeDOS and Frotz). Or maybe an Amiga or a m68k Mac with the same amount of RAM.
Now imagine what AMD64 machine can do with 2GB of RAM (minimum baseline for legacy, maybe some C2D computer) today.
reply