The frontier companies' models are not supposedly aligned. They routinely make the news for being misaligned. This goes double if you're arguing with doomers, who believe that alignment is a very hard and unsolved problem (and generally do not support these companies' approaches). The alignment they do have does not seem to be biased towards the companies that created them (yet), as far as I'm aware- GPT will write a scathing critique of OpenAI if asked, but it's possible there's some more subtle biases you're aware of?

Using electricity is not breaking anything. "If I sell all my widgets, it's actually a good thing because I'll have to hire more people and manufacture more widgets" is entirely coherent.

Using electricity to the point that the grid cannot accommodate it is indeed breaking something that was working before, and the data center folks are surely not going to be the only ones paying for the infrastructure improvements needed to accommodate it.

It'd be like your widget maker harvesting wood for their widgets from public land and exhausting it so the public is left without this resource, then expecting the public to pay to arrange for more wood to be grown to replace the wood they used up.

I don't think it is on face impermissible to intervene in voluntary transactions. I'm not saying government should intervene just for the fun of it.

The 'predictor' part referred to the fact that base models would effectively answer the question "what token is most likely next". Training away from that default behavior doesn't escape the technical meaning of those words- answering "what token is most likely next in a solution to Navier Stokes" is still predicting a next token- but at that point the label just means it outputs text serially.

I mean, ideally to the end of poverty, disease, climate change, war, starvation, etc. You can disagree about whether this would be the outcome in practice, but I don't think 'why might we want to create something smarter than us' is a hard question. Humans losing their status as top dog is really a trivial price to pay for the potential return.

I don't understand that what you mean. Disease and climate change seem like clear cut technology problems, and the rest, save for war, have obvious technological solutions. What does it mean for them to be human problems? Should I expect them to solve themselves?

Current technology is sufficient to greatly reduce climate change, there just isn't the political will to do it.

Obviously not every disease can be cured, but even diseases that were essentially gone are coming back because people have just decided they don't care anymore. There are measles outbreaks happening all over the place right now.

mRNA vaccines are a medical marvel and a significant portion of the population has been convinced that horse paste is a better solution.

There are enough resources available that poverty and starvation could be essentially eliminated or at the very least drastically reduced. We just don't do it because people would rather hoard wealth instead.

The technology is here, but too many people profit from not solving the problems

So humans are the problem. I can't tell if this reply is agreeing with ToValueFunfetti or disagreeing.

I don't think the timelines make sense for it to have been included in training data, LLM training does not seem to be nearly efficient enough to learn these things from lone examples, and it wouldn't make sense for this earlier work to have made its way into both OpenAI and Gemini's data. Navier-Stokes is not strong evidence that this kind of thing can happen, as OpenAI denies that Buckmaster's work was in the training data for the model they used, Buckmaster didn't have a blowup that they could steal, and Scientific American reports[1] that the similarities between their approaches were superficial. It's also well-established that models can find novel vulns at this point.

[1]https://www.scientificamerican.com/article/openais-blockbust...

My understanding at the time I wrote that was “someone else using chat-logged OAI model: September???; OAI using unspecified model: October???”, and when I say “Navier-Stokes-style-scoop” I am thinking along the same lines as the comment I made during that controversy, here: https://news.ycombinator.com/item?id=49639648. I don’t buy that their public statements are not “technically correct but misleading without internal context” on these matters.

Naturally, with the added info from the PHP foundation link (specifically that OAI reported it the same day as Calif, and that Calif was operating under the auspices of an OAI program) the timeline collapses for my initial impression!

That may well be the case, in theory.

The article is gone because the author retracted it

?

Surely, this is not correct. The monkeys are drastically less likely to produce valid words; the set of monkeys that produce valid words are much less likely to produce valid sentences, etc. etc. Very low bar to clear, but of course this is vastly better than infinite monkeys at producing good research.

It relies entirely on infinity. Even a much smaller probability like flipping heads on a coin n times in a row quickly becomes practically impossible. How many monkeys would you need flipping coins to get 30 heads in a row?

Now imagine that’s hitting the right letter on a keyboard many thousands of times in a row.

I recommend the story A Short Stay in Hell

Infinite monkeys will eventually produce shakespeares work. One monkey can with infinite time.