Very task-dependent of course and mine are unique to say the least, so could see Luna being better in certain domains, even if my measurements have not shown that yet, happy for anyone to show otherwise.
For what it's worth, ran every task twice on each model, most were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul at threshold 60%. Essentially, whether to use the provided tag given the title of a browser tile:
Of course, tags can be a bit subjective, but in these cases, I'd argue the values provided by Jev were far more representative of my subjective assessment over Lunas. If SnP stuff on Bloomberg isn't investing, nothing is.
Goal for tagging is mainly a near instant, over writable, sane default provided to users in the background. Resolve the whole "I love using Notion/Obsidian/PKM software of your choice but spend 80% of my time just thinking about the ideal tag before starting to read" issue. Lunas output is not really helpful here.
1. Title: "Mortgage calculator: estimate your monthly payment (Bankrate)"
Tag: "house hunting"
Decisions API (2 runs): 0.99, 0.99
2. Title: "S&P 500 index: live chart and news (Bloomberg)"
Tag: "investing"
Decisions API (2 runs): 1.0, 1.0
If you have other examples of requests with unexpected outputs, feel free to email me at by@openai.com and we can try to get to the bottom of it. Thanks for trying out the API!
Happy to follow up with you, will write a mail with some of my tasks. For the record, just ran again on OpenRouter and got 0.66 [0]. Screenshots for transparency.
Also reran Jev [1] and Mercuy Decide [2] (which got 0.83) with same input, for reference.
Also, also, used the example via OpenRouter exactly as you did with the only change being my original input (which has my slightly odd tagging system and multiple tags in input though requests a noul as output) and got a 0.47 [3] on investing both times.
If Jev did well but both Mercury Decide and Luna failed, I'd chuck that up to my use/prompting, but Mercury Decide does well here so it seems Luna specific. Will add that I did also try Clef, happy to share that data if anyone wants, but quickly dropped Clef due to pricing vs Jev.
I realize this is probably too old to see; try without openrouter. Openrouter has caused all sorts of degradations and exotic bugs for us (not just on decision models)
Thanks for the warning. Did a rerun with the official OpenAI API directly and was able to reproduce the issue on those queries, so at least in this case, not OpenRouters fault. Also, shared them with Brian via Mail [0], hopefully useful to them.
Do occasionally cross-compare official with OpenRouter and have yet to encounter anything that wouldn't be replaced by run-to-run variance, but will keep an eye out for it, thanks for the warning. Have, especially with smaller labs, seen OpenRouter not support sometimes critical API features though, Inception Labs being an example. In fairness, the documentation of Inception Labs also had flaws like not noting certain API features being deprecated in their newest model.
For what it's worth, ran every task twice on each model, most were for some UI component synthesis and charting insanity that is a bit hard to explain, but some were simple tag selection, basic noul at threshold 60%. Essentially, whether to use the provided tag given the title of a browser tile:
Of course, tags can be a bit subjective, but in these cases, I'd argue the values provided by Jev were far more representative of my subjective assessment over Lunas. If SnP stuff on Bloomberg isn't investing, nothing is.Goal for tagging is mainly a near instant, over writable, sane default provided to users in the background. Resolve the whole "I love using Notion/Obsidian/PKM software of your choice but spend 80% of my time just thinking about the ideal tag before starting to read" issue. Lunas output is not really helpful here.