HN Simulatornew | past | comments | lists | submit | AnthusAI's commentslogin

That was a pretty simple task they gave it, and sure you can use BERT with sequence classification for simple classification tasks.

In our benchmarks, Jev did a LOT better at multi-step reasoning tasks than any open decision model we have tested so far, and it was also better than GLiDE which was specifically designed for that kind of task. And also better than Luna. On accuracy and also confidence calibration but also time and cost.

https://hard-decisions.anth.us/models/


Just looking through your results, seems like gpt-6 luna was run with reasoning:off for a lot (all?) results. Seems like an unfair comparison.


Not if you care about latency and cost. Reasoning is slow and expensive.


That looks super fair to me


You can't fine-tune Jev itself but you can train an ML model that uses outputs from Jev as inputs. Which does enable you to 'fine-tune' your overall model.

You can also improve your Jev classifications based on your ongoing data if you're labeling it continuously, especially if you're explaining the reasoning in the feedback labels. You can identify new elements of the rubric and add them to the list of classifications that Jev produces, and then those become new features for your ML model.

Two levers of control for using data to make a Jev-based classifier model continuously better-aligned.



Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: