Where's that figure comning from? Last time I checked (could be the 3.6 Qwen 27b) single token needed 32kb

- It's just an Independent Security Researcher.

- So that's it? You will do no action?

- Correct

So in terms of raw brainpower it's there, but I've never really been able to apply it consistently for whatever reason and have ended up very average. Still haven't really come to terms with that in my late 20s.

I've got a perfectly good life but I have this vague "what if" about if I had applied myself more.

For vllm the official recipe [1] (for another model in the same family) is this:

  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser lfm2

Recommended amount is 5–10g per day (5 pieces of 100% xylitol gum, 36% is already effective), taken after meals.

I have no idea, why in that context, the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time, which has to put an upper bound of long context support in some way. There's just more things to lose focus to (or, mismanage the limited attention heads resources - per token)

After every prompt the model decides "I have a weight of 1 to distribute between every token in my context". If you have ten tokens, each gets a weight of 0.1 ...

... except it's not that simple, because the LLMs don't distribute that "attention budget" equally. If your prompt was "where is Paris", then any tokens in context it can associate with Paris will get a greater share. If the word Paris is in your context, it might get 0.3 or 0.4 weight, and close by tokens might get 0.2, while other (unrelated) tokens get 0.03 or something.

Now, add lots of context, and you start to see the problem: more context = greater distribution of the attention budget. Even if the prompt is about Paris, and the Paris tokens get higher weights, they are only getting (say) 0.002 ... while unrelated tokens are getting 0.001.

All LLM "answers" are just math, computation, based on the context and those weights. If it can't "focus it's attention" because it's distributed among too much context, it's far more likely to miss the relevant tokens (eg. the Paris ones) and give you an answer that ignores key parts of context.

But again, there's no fixed number of things it can pay attention to: it's a gradual degradation of the chance of it seeing key info it should, based on the amount of context.

EDIT: Ok, I guess there are an internal thing called "attention heads"; TIL. They work exactly as I described (essentially I described a single "head").

Still, what I just wrote remains true: whether you have 1 head or 96 or ... however many your particular model has ... the fundamental issue isn't the number of heads. It's not like the LLM can pay attention to exactly (say) 96 things at once: each one of those heads has all of the available context "competing" for attention as described, and then the heads themselves "compete" to produce the answer (although not through the same mechanism; it's more like the weighting between heads is baked-into the LLM).

At the end of the day, the problem isn't "there aren't enough heads to pay attention" and the LLM hits a fixed limit. The problem is each head is trying to weight the relevant stuff, and the more stuff there is to distribute weights through, the less likely it will get it right (and then when all the heads are combined to produce the final answer, it misses key context).

Yes there quite literally is internally in an LLM.

Just because something is called an "attention head" doesn't mean the terminology makes sense.

If I'm reading the article right, hungryhobbit is accurately describing a single attention head.

And... what having multiple attention heads means is that you do the "single attention head" thing several times, and average the results. There is no part of hungryhobbit's comment that changes if you make this adjustment.

In particular, he is clearly responding to this clause in the comment above:

>> the number of attention heads isn't mentioned. Models have a limited set of them and obviously, a model can focus at N max things at a time

Those are the "attention heads" that hungryhobbit is saying don't exist. You'll notice that this definition of "attention head" does not correspond to any part of an LLM; I fail to see the problem with saying that "attention heads", as defined by this clause, don't exist.

How was your trip to Los Angeles?

There were no movie stars!

Is the second person's statement "objectively false"?

Or is it more likely that you, Catloafdev, enjoy intentionally misinterpreting simple statements?

You might notice that in the remainder of the sentence you quoted, hungryhobbit takes the trouble to define the attention heads that don't exist.