Always exuded so much authority & confidence. What a legend.
But he start losing his hair at 17 and really struggled with it. A turning point came when a friend (a Hungarian judo black belt) and his wife wrestled him down and physically cut off his toupée with scissors.
His agent even had him do the Star Trek audition in a wig, but luckily the crew had better ideas.
Aside: Does anyone else find the new AI texture explosions Apple's using on their marketing pages now a little unsettling and un-Apple? I guess I'm desperate for at least one tech brand to feel / remain human.
This has been my hope for a while! I live in a 1911 house and the amount of small detail everywhere would be impossibly expensive by modern construction standards. I also like modern minimalism, but I know many of the drywall corners everywhere in modern homes aren't design decisions as much as cost savings measures.
Which benchmark? Would love to see it & any datasets.
It’s super tricky to tease out whether the similarities point towards distillation. A lot of the aesthetic similarities are naturally emergent from the training data.
The conversation examples are impressive but this made me say holy shit out loud:
"it runs locally on a MacBook Pro M4 Max or better. And we figured out how to train Dessa on only one NVIDIA GB200 NVL72."
If it really can replicate that low latency + that expressiveness on local devices, that's one heck of an accomplishment for "indie" AI.
> > one NVIDIA GB200 NVL72
That’s around ~$4-6 million in hardware cost (one rack, 72 GPUs + accompanying hardware).
that’s a good clarification, skimming some of the comments here I was thinking they trained it on a single GB200 gpu. THAT would be a huge win for indie AI
Edit: Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems. The claim of “we trained this on a single rack of gpus” is definitely much less impressive than “we trained this on a single gpu”
> Nvidia GB200 does not refer to a single gpu, Nvidia’s naming scheme is very confusing. Considering past gpus were called H100, H200 etc, I thought GB200 would just be a newer GPU, but it does actually refer to their 30+ gpu rack scale systems.
It's even more confusing than that :)
GB200 = 1 CPU + 2 B200 GPUs (packaged together)
NV72 = A rack of 72 B200 GPUs (or 32 GB200 superchips)
Think of GB200 as something like Apple's 'Ultra' chip. Not exactly, but similar. They fuse the chips together with a very fast interconnect (connection).
Also a speech to speech model it can't do any tool use or you can't guide it much am I mistaken? How will it be used for agentic uses is the part I don't get. Edit: this is response from Argon if anyone was also curious: You are not mistaken about Dessa: because it was trained solely on raw audio with zero text data, it cannot read text system prompts or output code for tool calls on its own. To use it as an agent, developers must either steer it purely through spoken voice instructions or pair it as a conversational "voice front-end" alongside a separate text LLM that listens to the audio and executes the tools in the background.
> they didn’t have to worry about hosting/billing/subscriptions or any of that yet
Or marketing, which they kinda get for free. So much research goes unseen, there’s huge competition for attention, and most research is very limited in that regard
I would say that’s the biggest factor that makes the team “not indie”
At the same time, like you said, still amazing accomplishment, very cool model, and great work
Speech is in quite a weird spot when it comes to locally running it. Some of the best STT and TTS models run at faster-than-realtime speed on decent consumer hardware with quite small models (so also no huge RAM requirements), but the tooling and library ecosystem is much more fragmented, making it a lot harder to build actual working products around it. There is also a much more noticeable coverage / quality gap when it comes to languages other than English.
As a speech-to-speech model Dessa is of course in a different ballpark, but given the intelligence it exhibits, it seems to be working with not that many parameters. The developers also seem to be admitting that to some degree ("in fairness, LLMs also have to learn C++ and whatever the hell Angular is"). I'm wondering what the utility of this would be. Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
But the thing that makes this special, is explicitly cutting things out that make LLM's capable of that. As gpm's sibling comment points out, if you would add that back in, you would just end up in the converging to the same place (with the same downsides).
The only way around that would be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
> be to somehow get all world knowledge and tool calls into a audio space, which seems like quite the stretch.
It seems quite easy? They have the natural language bit nailed, they just need to convert the text with it. And it wouldn't be hard to extend architecture this into outputting tool calls.
This thing has the "the natural language nailed" to the same degree that a BERT has. That's a far cry from the intelligence level required to output text of any sensible length, and even farther from the structural adherence required for tool calls.
Highly-structured outputs (like what you'd need for tool calls) seems like it'd be easier to put out than arbitrary text, no? At the very least, even something like BERT could make the decision of “Would any of my available tools be useful here?” and delegate to some deeper-thinking model to structure that tool call.
> Given that it lacks both broad world knowledge and tool calling integration in any way, that seems kind of like a dead-end.
In its current form probably. I think the key to taking this further would be to split the listening model from the speaking model, and have some model(s) in between that can do those database/knowledgebase queries and tool calls and such and influence how the speaking model response to what the hearing model perceives. That's probably pretty similar to how ChatGPT Live and similar are operating (STT→LLM→TTS), but Dessa demonstrates that it doesn't have to necessarily be text/language, specifically, as the intermediary — and therefore it doesn't have to necessarily be an LLM, specifically, as that intermediate model.
>> I'm wondering what the utility of this would be.
Exactly, it does feel next level compared to STT -> LLM -> TTS setups, but unless you can steer it effectively to do something like helping a customer do a complex task it is just a chatbot?
I can't even imagine what things like RAG, system prompts and other context setups would look like with this.
What STT and TTS models do you recommend that would run well on consumer hardware?
I use Kokoro 82M and it's good overall, but it does make mistakes on some common phrases, and it's not real time fast (runs at about 6-7X content-to-encoding time ratio on my home server)
I don't know whether people are just ignorant but it is heavily likely say the S2S without any intelligence can run on a pro-sumer device. Just because it is "OpenAI" and "Google" and not public knowledge or will be open source/public weights doesn't mean you can't reasonably infer.
Probably Dessa started those demo conversations with some context, so maybe it could receive documents and speak about them. If those were random conversations started with no context, the people at Block probably recorded a lot of uninteresting chat and cherry picked the few good ones.
Current tool calling just parses llm output for flags that indicate a tool should be used.
How would you do that when output is raw audio buffers. You would need your conditionals to be like: if (audioDataArraySeemsLikeAnEmail(llmResponse)) {}
This isn’t even the sandbox. It needs Docker or other virtualization. This seems to sit a level up and orchestrate sandboxes and access to network & resources. It’s a pretty heavy setup from what I can tell.
But he start losing his hair at 17 and really struggled with it. A turning point came when a friend (a Hungarian judo black belt) and his wife wrestled him down and physically cut off his toupée with scissors.
His agent even had him do the Star Trek audition in a wig, but luckily the crew had better ideas.
reply