this is really cool. i especially like the gesture mapping... using the distance between your hands for volume, their average height for pitch, and the angle between them for vibrato feels unusually intuitive. the fact that those dimensions can move independently makes this feel like it could support a lot more than a theremin sound
they're LLM-generated, web-embeddable synths, and the real-time controllable parameters can be defined in the prompt — so your hand tracking could map directly onto whatever controls make sense for the instrument
Interesting product. Are you really using Fable 5 to generate sounds?
FYI, in Chrome for Android, the only way to hide those bottom sheets is to tap the handle. Grabbing and sliding them down, reloads the page, which is a little annoying.
I created a gamelan sequencer with it. I intended on using samples but it went ahead and created a bunch of synthesized instruments in javascript with the right (ish) overtones, in the right tuning system, it was pretty crazy. 3d instruments animation with 3d audio positioning.
I’m exploring an analogous idea for music. The quality of the musical output has improved noticeably with the latest frontier models
Karpathy’s point about the models not being able to easily audit their work is something I’m struggling with —- how can the audio output be made perceivable? Curious what people think about this question.
also - i made you some sounds with Underscore:
chill / evolving harmonic thing: https://underscore.audio/s/cmp_f2b79a32-57e/chord_lantern
chaotic / glitchier thing: https://underscore.audio/s/cmp_6abde2c8-2d9/morseMind
they're LLM-generated, web-embeddable synths, and the real-time controllable parameters can be defined in the prompt — so your hand tracking could map directly onto whatever controls make sense for the instrument
you can also generate more structured compositions like this: https://underscore.audio/s/cmp_9175f7d2-fe0/drifter_glimmer
happy to help you wire one up / iterate on some sounds if you're interested
disclaimer: i'm building Underscore :)