Unless that takes 700 attempts on average I don't think that actually works.

That assumes each bit is a coinflip, doesn't it?

Even Markov chain autocorrect tools do better than 50% odds*, and even GPT-2 was significantly better than that kind of autocorrect.

* at the word level; IDK how redundant/efficient language is when it comes to bits-worth-of-fact-claims-per-word. But "your cat is sitting on my" -> [mat, laundry, roof, head, belly, laptop, microwave, …] clearly has many bits of information, and a Markov chain will encode the most likely next word even if the user doesn't know what the most likely next word is. Verifying where the cat is sitting is also very easy, as is correction.

I assumed a coin flip, indeed in practice you would likely achieve far less than 1 bit per attempt.

Far more than 1 bit for most attempts: they need that just to be able to write coherent sentences, and a lot more to be coherent sentences on the right topic.

Some specific conclusions would be far less than 1 bit.

The average will depend on both the question and the AI.

This is about information from the sender to the recipient. It is fundamentally impossible to transfer more than one bit in one binary decision.

A LLM adds noise, not information. At least in this framing.

Frankly, it's a stupid framing.

LLM is not a random symbol generator (hint: training data is not random), and no reasonable person is going to just prompt an LLM and send its output without giving it at least cursory check (at the very least so that blatantly stupid hallucinations don't paint the sender as inconsiderate or incompetent).

That check alone can add bits to the final signal.

I'm not sure we're working in the same framing here. For one LLMs are random, sure you can fix the seed but you don't have to, you can even replace the RNG with true random noise.

So there are 2^300 possible ideas, only 2 outcomes from the cursory check, how do you get 2^700 outcomes? Most of those are just random variations the LLM added which is not a transfer of information. You would be lucky to even identify which of the 2^300 ideas was being conferred.

"Random" is too loose a word, but they were responding in a context where it meant coin flips.

LLMs are not even odds on all possible outputs, they are biased towards patterns which are upvoted by the training mechanism (at a minimum: the source material, RLHF, and synthetic data).

The information any trained model transfers to output, is information it gained during its training.

No single human is capable of having consumed all that training data.

There comes a point where someone isn't so much talking to the sender as having an unsolicited AI chat. Especially when most of the information didn't come from the sender in the first place.

It's actually hard to define the difference between information that comes from the model and just random variation, maybe something to do with the cross entropy between the sender and the model?

> There comes a point where someone isn't so much talking to the sender as having an unsolicited AI chat. Especially when most of the information didn't come from the sender in the first place.

Indeed.

The best case is a P vs NP situation: can the claims from the AI be easily verified, or not?

This does not excuse people too lazy (or overly impressed*) who fail to attempt the verification.

> It's actually hard to define the difference between information that comes from the model and just random variation, maybe something to do with the cross entropy between the sender and the model?

Mm.

Thanks to a philosophy course I did half a lifetime ago, I think there's a fundamental problem defining "information" in this context. It feels like it should mean "knowledge" because the discussions about Shannon entropy and transmission channels assumes there is an actual source-of-truth, but my conclusion from discussions about why "knowledge" can't just mean a "justified true belief" is thay I now don't believe we can do better than "belief"; an LLM can generate tokens that change your beliefs, but ultimately neither you nor I nor some annoying colleage who has made themselves redundant to the LLM, can be an oracle with definitely-true knowledge.

(I have of course tried asking an LLM about this thread; I don't feel it illuminated anything new for me, none of what it suggested made it into this comment).

* In the early days of LLMs, I was overly-impressed. Then I realised we were doing the same thing with LLMs today that we did with 3D graphics in the 90s, where every new engine was hailed as "photorealistic" only to be dismissed 6 months later when something better came along: https://archive.org/details/nextgen-issue-26

Only now it's every 11 weeks rather than 6 months.

> and no reasonable person is going to just prompt an LLM and send its output without giving it at least cursory check

I'm reminded of an old quote:

  The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.

Touché.

In information theory, each bit is a coin flip by definition

I realise I phrased this poorly.

I will try harder. Consider entropy.

The first sentence in this comment contains 32 characters; from the point of view of a naïve channel with no compression, that's 256 bits (given none require breaking out of the first bytes of UTF-8).

It did not take 2^256 attempts to construct the first sentence in this comment, because the generation process was not flipping coins per bit.

LLMs also do not emit bits chosen with a [0: 0.5, 1: 0.5] probability distribution.

From a compression point of view, the bits-transmitted-per-bits-in-message ratio can be reduced such that more likely messages use fewer bits than less likely messages. However, this requires the receiver to agree with the sender what the probability distribution over tokens is.

Intelligence is, amongst other things, a compression algorithm. If I can predict your next token, and we both know this, we can agree in advance that you don't need to actually send it.

No single human brain is able to predict the output of an LLM anything like well enough to do that.

In entropy terms: LLMs are noisy sources, their output does contain false statements, yet they add more bits of signal than of noise relative to a human alone.

Or at least, they can add more add more bits of signal than of noise relative to a human alone, but humans who blindly copy-paste the output of an LLM without checking are a pain and add zero value to whatever situation they happen to be in.

For some hypothetical scenario, writing software because I know they can do that, asking an LLM to write some code for you may easily give you 10 kilobits of positive information (code that mostly works), and -30 bits of noise (each bit being one binary decision's worth of incorrect choice by the LLM in what to write, i.e. bugs); if you as a user don't know how to handle the -30 noise that could easily be a totally useless app, but if you can filter out 30 bits of noise, either manually because those 30 bits happen to be your skill set, or even in some cases by prompting it again with the failure mode, then you get to benefit from the 10 kilobits of good stuff that you didn't have before.

In many (but not all) cases, LLMs can fix more than 1 bit of mistakes per follow-up prompt.