I used to think byte level tokenization was the answer, but humans also think at a word level and only reevaluate the words at a character level when asked. The solution to better tokenization across languages is likely to be learned tokenization. Here is one attempt I have seen: https://github.com/SamD770/bitter-lesson-tokenization
Careful: the problem is very certainly ___not___ counting letters. That is only a telling way to check "is the NN checking or not?". We demand that NNs for consultancy tasks check, strictly.
I'd approach this distinction differently. Prediction from compression is valid within the distribution of the compressed data. Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
> Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
This is a great experiment. Do some world building offline about Smergle the Keen finding the Hoopla of Kynashiusous. Then ask ChatGPT what reagents would have been more effective in breaking through the borantiastic door than squeeble stomach juices.
Of course we know the answer - it will either drop into a 'I have only been trained on data before blah blah blah' or 'I cannot talk about current world leaders' or 'I have not been trained on your hackneyed and unpublished romantasy saga, why don't you tell me about it?'
On the other hand, if it is aware of the general concepts of your concept, and has bridging concepts for related topics it has been trained upon, it might try to apply the general concepts to a specific but different scenario.
I’d argue the opposite. The geometry involved in reconciling photos from two separate lenses with different focal lengths is quite challenging. If you pay close attention you will see some differences in the before/after samples in the site (for example the ear in that kitchen selfie).
The math is surely more complex than aligning same-lens photography even when you consider the change in angle and perspective between two shots from the same lens.
Not really. Whether you have two different lenses or the same lenses with some offset from hand movement you need to accurately model the lens(es). Once you can do that for one lens you can just as easily do it for two - it's just more variables to optimize, not a fundamentally harder problem. Panorama software has supported images from different lenses since forever.
reply