I've observed the "let's implement this text analysis task with regex" quite frequently since the early days, which does lean toward supporting the "AI doesn't know about AI" theory.

I think a lot of it comes down to game design is hard.

There's a lot of moving parts that go into it, and even experienced teams have difficulty finding the fun. It's extremely common that games that are now popular and beloved weren't very fun until a few months before they released.

> It's extremely common that games that are now popular and beloved weren't very fun until a few months before they released.

I'm not sure that's so common, most games released that weren't fun before they were all completely finish, don't fare so well in the end. What tends to do well, is stuff that is fun/feels good early on, but might not be "visually/creatively complete" until just a few weeks before launch. That last part, is extremely common in game development, maybe you got the two confused.

Never happen...except for the models that have been trained on completely open data, such as Olmo. They're not frontier class, but they do exist.

And even then, you have just the word of the producer that there wasn't any poisoned data used for training. For that you would need "reproducable builds" (i.e., training). Which makes no (economic) sense, given that the training is the expensive part.

There is definitely a kind of monocrop problem in some fields, where it seems like having a very diverse spread of academic investigation is needed to have enough diverse traces through the search space. And globalization has been flattening that.

AI has been turning computer science into biology for the past decade or so. By which I mean that things like neural networks need to be investigated empirically, constructing methodologies and instruments that more closely resemble how fields like biology and medicine have to probe the very complex and messy reality that is beyond our current capacity to fully express in symbolic precision.

Now math gets to deal with that same reckoning. They were already well on their way there with previous Lean proofs, but this has pushed things beyond that horizon and I'm not sure some of the mathematicians are ready for it.

Science has always had an empirical component separate from its theoretical one. For a long time in human history, science was mostly empirical. The periodic table is a great example of mostly empirical observation organized into a pattern. I think the science of the 20th century was the "triumph of theory" so many of us have forgotten what a more empirically driven STEM world is like.

Agreed, I keep thinking to myself that there's a huge mathematical question right in front of us today that is in exactly this vein - all the various nuances of why LLM's work so well is a mathematical question. As far as I know, it's not really understood beyond "we do this basic thing (that makes sense) to predict that a noun is followed by a verb, and then we scale it up a bazillion fold and it can contribute to mathematics research".

As a comparison, classical computing has been scaled up a bazillion fold too, and can do things which are absolutely miraculous, but every layer of abstraction is discretely understandable.

To be clear: it’s not for lack of trying.

I'd assume that theoretical math prides itself in trying to be as close to pure logic as it gets. It has very little, if any empirical elements (at least that's what it looks like to me). And I understand the value in trying to preserve that pure logic element. In the end, biology is just chemistry, chemistry is just physics, and physics is just math. If you feel like your profession is the bedrock of all other science, it probably feels weird to shake its theoretical foundation.

While studying neural networks empirically like biology is one possible approach, there's no reason for that to be the blessed approach other than a combination of inertia and current lack of understanding. It's been only 15 years since AlexNet. Scientists struggled for centuries to model atoms before developing quantum mechanics and later QFT as an accurate quantitative framework.

Also, while biological systems simply exist in nature, artificial neural networks are ultimately mathematical objects with various properties that have yet to be uncovered.

I really like this way of putting it

Because Yahoo in particular was very good at destroying goldmines shortly before they became ultra valuable.

There's a number of places that destroyed vast amounts of data in the wind down of ZIRP that probably regret it now.

The early pre-United States had pamphlets and newsletters. Like, a lot of them. It was a big influence on the early post office. You could get rich running a printing press but you did it by printing for the masses.

The yellow journalism era, on the other hand, was a little closer to the rich owning the press.

The penny press, and a ton of those pamphlets amount to what we now call tabloids. They were funded by advertising too, and used fear & outrage to sell copies. Led to some pretty funny conspiracies like The Moon Hoax (https://en.wikipedia.org/wiki/Great_Moon_Hoax)

Times are not as different as we'd like to believe

I wonder if it partially because "write like a human" is kind of a vacuous request. Like, it's the objective everyone including me has been saying that we want, but there's no one way to write like a human and and humans don't even have a good definition past "I know it when I see it."

There's a lot of work in the humanities about different aspects of good writing, but that's not quite the same thing. And anyway they tend to assume a pre-existing level of writing ability. Students are supposed to learn good writing through practice; there are rules and exercises but they're incomplete.

I think as much it is that people write by grappling for the right phrase to represent some inner feeling or concept, writing in part for themselves, whereas LLMs write always and only for an audience.

It’s much easier to understand this once you think about other generative forms. MidJourney never just sits down and draws for fun, so fun never informs its art (only the outward appearance of others’ fun, separate from the fun itself). Suno doesn’t waste hours trying to find riffs on a guitar, so its output is never informed by the direct joy of getting it right. Its music is never optimised for playability on a particular guitar with a scratchy seventh fret and a too-high action. Neither Midjourney nor Suno have evolved their styles due to short-sightedness or carpal tunnel.

If you had a human writer who over a long career only ever wrote articles from an outline given to them by someone else, and you had all the outlines and all the resulting articles from those outlines, and you could train an LLM to generate an article from an outline, it still would not be kicking itself frustrated by an inelegant phrase in a prior article, it would not avoid certain phrases out of a passive aggressive reaction to some editor’s note, it would not ever just rush an article because everyone is gathering at the pub, and it would not choose an analogy just to rub the author of a bitchy critical letter to the editor the wrong way. An LLM could not “subtweet”. It could not write a series of articles hoping one important person will spot that they are auditioning for a job.

Creators have unseen, undocumented influences and motivations that inform their work over a long period. I don’t mean to say that these individual influences can be reliably detected in individual pieces of work. I do mean to say that I think their broad absence tends to be felt in LLM writing. As readers we develop an affinity for writers as much as for their writing, and we do this in part because we deduce things about them.

Very elegantly put! So much so in fact, that I finally put down my phone to grab my laptop in order to write this. I have had a similar much cruder version of this thought: LLMs are not human, they don't know what it's like to stub their toe, despite having read probably millions descriptions of it. Even amongst humans there are experiences which are impossible to share, despite us having evolved to communicate quite effectively. Writing is just a small part of it, and cannot replace the actual experience of being human.

Each person writes in a different personal way, so writing “like a human” would actually require a model being able to purposefully make the specific choices that an individual human writer does.

However, general purpose LLMs like Fable have been trained on huge amounts of all kinds of data, and therefore find it exceedingly hard to break out of the grooves carved by that data. They can’t avoid defaulting to centroids and averages, even when they are trying not to. This makes it possible for classifiers like Pangram to discriminate their writing.

A plausible way to work around this limitation would be to train a LLM on a limited and cohesive subset of writing materials, so it would absorb their specific writing style.

One example might be Talkie, a LLM trained on pre-1930’s English text. Talkie is a far smaller and less powerful model than Fable.

And yet, Talkie’s writing is so distinctive that it is often classified as human by Pangram.

Is pangram using knowledge about specific models? Could you throw it off by having several models generate a paragraph each or so?