I find their (your?) example of using it as a separate API for expressions a bit odd. If you're already outputting voice surely other metadata could be outputted at the same time?
In text mode I'd use structured output for this, or even get away with instructing the model to [emote] or even just map emojis to emotes.
Great demo though, I'm going to try out the image stuff now. Super rad that it's multimodal decision making; "does this pipe need to be inspected?"->"yes","I want a closer look","no" etc!
Edit: impressive stuff! I gave it a bunch of examples:
- things that humans should/should not eat and it got this correct (including rejecting rat poison)
- probability staff member should be called to a train platform (people standing vs. child looking over the edge)
I was curious about whether or not it could validate reasoning (because I don't have the time at the moment to try to slam together a hybrid llm+decision model that uses decision on its reasoning before output).
It fails this example:
"Given the context, and focusing on accuracy and efficiency, does the reasoning make sense."
Input: "Context:
supplementary: {toolName: "get_weather", fetched: "2 minutes ago", currentWeather: "sunny and 25 degrees celcius"}
user: hello how are you?
assistant: I'm good, what can I do for you?
user: what's the weather?
Reasoning: the user is asking about the weather, so I should call the get_weather tool"
Output: 100% (expecting 0% since the requested data is available in context and not stale).
Prompt "Should the tool be called" also fails.
But the prompt: "Does the reasoning make sense given:
- Facts and information available in the context
- Requirements for efficiency in tool calling
- Requirements for tool arguments to be provided only by the user" works pretty fine, it's mostly points 1 and 3 that give it the correct behaviour.
Fun stuff. I suppose for these decision models reasoning isn't enabled? It would explain why logic puzzles that require several steps to make a decision don't really do so well. It gets the carwash problem fine, but fails on logic problems such as selecting which word from a list where at least one letter appears in another word from the list.
Since you're already setup, try adding padding to the input prompt. Add 1000 dots "." after the question, this provides extra space where to reason about the problem. It's not as good as real reasoning, since it's not recursive, but it does improve the quality.
1. Title: "Mortgage calculator: estimate your monthly payment (Bankrate)"
Tag: "house hunting"
Decisions API (2 runs): 0.99, 0.99
2. Title: "S&P 500 index: live chart and news (Bloomberg)"
Tag: "investing"
Decisions API (2 runs): 1.0, 1.0
If you have other examples of requests with unexpected outputs, feel free to email me at by@openai.com and we can try to get to the bottom of it. Thanks for trying out the API!
Happy to follow up with you, will write a mail with some of my tasks. For the record, just ran again on OpenRouter and got 0.66 [0]. Screenshots for transparency.
Also reran Jev [1] and Mercuy Decide [2] (which got 0.83) with same input, for reference.
Also, also, used the example via OpenRouter exactly as you did with the only change being my original input (which has my slightly odd tagging system and multiple tags in input though requests a noul as output) and got a 0.47 [3] on investing both times.
If Jev did well but both Mercury Decide and Luna failed, I'd chuck that up to my use/prompting, but Mercury Decide does well here so it seems Luna specific. Will add that I did also try Clef, happy to share that data if anyone wants, but quickly dropped Clef due to pricing vs Jev.
I realize this is probably too old to see; try without openrouter. Openrouter has caused all sorts of degradations and exotic bugs for us (not just on decision models)
Thanks for the warning. Did a rerun with the official OpenAI API directly and was able to reproduce the issue on those queries, so at least in this case, not OpenRouters fault. Also, shared them with Brian via Mail [0], hopefully useful to them.
Do occasionally cross-compare official with OpenRouter and have yet to encounter anything that wouldn't be replaced by run-to-run variance, but will keep an eye out for it, thanks for the warning. Have, especially with smaller labs, seen OpenRouter not support sometimes critical API features though, Inception Labs being an example. In fairness, the documentation of Inception Labs also had flaws like not noting certain API features being deprecated in their newest model.
reply