Can someone do a ELI5A of how they achieve classification over any user defined list of items? Normal neural networks do a softmax over a known output set to get probabilities
You can achieve open-vocabulary classification by making the final weights in the softmax come from a category encoder instead of being fixed learned weights. So instead of
softmax(encode(input)*learned_weights)
You have
softmax(encode(input)*encode(categories))
I'm not sure if Jev does it this way, but it's how you get open-vocabulary zero-shot image classification with models like CLIP [1].
I can think of two possible approaches
1. Jev limits to 255 distinct options. So they can preprocess your set of options and “tell” the LLM via input tokens 1 = red, 2 = blue, etc then jev need only output softmax over 255 states while benefiting from pretrain of other LLMs
2. You allow the forward pass to output over the total token state but mask over the logits to limit to the user options. Less plausible? bc tricky when input is multi token which they clearly support.
My guess would be option 1. Didn’t read the kev repo here which would also explain
An raw LLM no, but given code execution it can do a good job. I had my Claude Sonnet write a sonnet in this form after 14 rounds of iteration with Python:
The autumn wind moves slow across the field,
and every falling leaf now yields its fight.
The summer gold at last has ceased to yield,
and short days now give way to longer night.
I walk alone beside a calm and quiet stream,
and watch it slowly carry every fallen leaf.
I think of you as some half-forgotten dream,
and taste again that same old, bitter grief.
The silent stars come out to watch it still,
and pale moonlight falls gently on the hill.
A lonesome owl calls out from past the mill,
and time moves on, unhurried and calm, until
the dawn returns to paint the sky brand new,
and I still recall the love I lost with you.
As a corollary, the output classes can be any set, rather than needing to be set before training.
reply