my experience with LLMs is that when you get answers to questions you didn't ask that it's repeating something that already exists.
In other words, it probably regenerated a tool from which it was trained on.
You could train an LLM on a dataset that intentionally excludes certain data, and see if it is able to extrapolate outside of its dataset and come up with those excluded items independently.
For example, remove all code from your training data, and then see if the model can code regardless.
Expecting to go from having seen no code to being able to code within a context window is a pretty high bar. More useful would be asking for code that does something you're pretty confident no-one has done before.
(TBH, when people claim this, I do wish there would at least be occasionally an actual pointer to something that was copied. I understand it's probably not going to be doable in all cases, but without examples it sure feels like a weak statement. In my experience the recent models are good at doing things I'm pretty sure there is not a close reference for in the training data, though for most part I wouldn't classify them as particularly difficult tasks either)
Thanks for the elaboration. (Though I'm not sure
dmarinus would agree with your criterion?)
> For example, remove all code from your training data, and then see if the model can code regardless.
Would you accept this weaker version: make up a new programming language (that's guaranteed not to be in the training set), and see if the model (which has trained on programming in existing languages) can cope?
