I switched to a decision model for my local "AI" voice control thingy. faster-whisper produces the text then onnx scores all my HA entities to determine which one I'm talking about .. then it lists all its actions and a second scoring round to determine what action I want. some regex to extract numbers if I'm saying things like "water the lawn for 10 minutes". It's pretty dumb when compared to what an LLM can do but considering you now have semantic scoring "at your finger tips" without resorting to levenshteins or other crude algorithms like that it's insane how smart it can look.
And to answer the obvious question: because latency. This thing turns on the light or water valve in under 100ms on a laptop from 2001 with 8gb ram .. all running locally on the laptop.
Consider looking at ChatScript for these kind of formulaic language matching -> variable extraction. It’s deterministic and instantaneous compared to any LLM
This is amazing! We are also fascinated by how fast Jev is. It's very much a breath of fresh air in the age when LLMs tend to take a much longer time and reason about tasks.
under 100ms on a 2001 laptop answers the why-not-llm question. what happens when the top two entities score close together on a short command like lights off, do you just take the first or ask back?
And to answer the obvious question: because latency. This thing turns on the light or water valve in under 100ms on a laptop from 2001 with 8gb ram .. all running locally on the laptop.