This all seems precedented on model capabilities that came into play over the last 3-6 months. How do you establish this new paradigm when we don’t know what the models will be like in 1,2, 10(?!) years from now.
The big deal is speed (high) and cost (low). This week I’m messing around augmenting mouse and keyboard interactions in an application with Deepgram + System 1 (Jev).
You can talk to an application and have it respond in real time with this combo. It’s clear this kind of general purpose intelligence may be a new development primitive.
However, at this stage, it’s difficult to work with for a few reasons. It’s API only, and you have to shape calling software to the way it communicates.
It’s not clear yet if what we’re missing is a new programming language, or some kind of harness or tool over the capabilities. LLMs were like this early on as well until better harnesses came along and reduced friction in their use.
Down the road, I highly suspect we’ll see:
- Intelligent context assembly using System 1 that summons memories as needed in LLM conversations and handles simple commands.
- System 1 programs that run in the datacenter and reach out to the request initiator on specific instructions like a CPU that has hit a memory barrier.
To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.
And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.
But I still need to experiment with either Jev/SemIf myself.
> To get calibrated probabilities sounds like a very good feature, if they are indeed well calibrated.
This is a valid point.
> And in my experiments even Qwen 3.8 has a hard time to consistenly conform to a schema, requiring retries, JSON cleanup etc, so to have a model of similar quality (SemIf et al) that simply cannot deviate from the schema by construction could be very helpful.
Literally every inference framework supports constrained encoding. You can make the model choose only from allowed tokens and you can infer only the first diverging token.
It's baffling to me that no inference provider actually exposes this functionality, so you have to run the model yourself to do it.
Given these numbers it has some potential to become quite usable for unattended workloads, especially if decode can be batched across multiple sessions (ideally enough of them to get some reuse of the sparsely streamed weights). (Of course this ultimately makes prefill times explode as you try and increase the workload even further. But that's arguably the natural bottleneck on any interesting local LLM inference, being a compute bound step.)
idk I thought the same until I got personal amex support at work, now it’s my roofline on how useful/trusted an agent could be. Now I book all my travel through them, and it’s an absolute blessing
I’m quite happy that my friends have labelled me as “an absolute idiot with anything but computers.” It’s much easier to relate to people at that level, adopt a happy-go-lucky attitude, and being wrong about 99% of things. being the smart one isn’t such a great label, and is (mostly) all in your head
