What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?
The LLM right now tells a much lower level neural network controlling the legs, arms and head what and how to move at a ~ 500Hz freq, which controls thing like the stumbling. All motor joints and IMU inputs are given to an ONNX-based RL-trained NN which then outputs motor movements (scary since its works well for the scenarios where its has been trained, but it goes in unexpected ways for untrained scenarios).
In any case latency is acceptable for navigation I guess, but avoiding moving obstacles is definitely a no-go right now.
Thanks! This is encouraging. I try to use Claude Code for producing client facing presentations that are static html files with charts, tables, and annotations. It never gets the tone correct and phrases things so weirdly - it drives me mad. I have to really fight it to stop it writing insights in a flowery and verbose way
What exactly makes it proactive? I created a similar system that ran on a ticker, but then could also set itself to run in n seconds in the future. RSI came from the actions available to the system and granting the ability to modify itself. The prompt was a string of messages made from static and dynamically generated sections (like memories or plans of tasks or outputs from actions taken on the previous turn). It worked really well
What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?