That was a pretty simple task they gave it, and sure you can use BERT with sequence classification for simple classification tasks.
In our benchmarks, Jev did a LOT better at multi-step reasoning tasks than any open decision model we have tested so far, and it was also better than GLiDE which was specifically designed for that kind of task. And also better than Luna. On accuracy and also confidence calibration but also time and cost.
You can't fine-tune Jev itself but you can train an ML model that uses outputs from Jev as inputs. Which does enable you to 'fine-tune' your overall model.
You can also improve your Jev classifications based on your ongoing data if you're labeling it continuously, especially if you're explaining the reasoning in the feedback labels. You can identify new elements of the rubric and add them to the list of classifications that Jev produces, and then those become new features for your ML model.
Two levers of control for using data to make a Jev-based classifier model continuously better-aligned.
In our benchmarks, Jev did a LOT better at multi-step reasoning tasks than any open decision model we have tested so far, and it was also better than GLiDE which was specifically designed for that kind of task. And also better than Luna. On accuracy and also confidence calibration but also time and cost.
https://hard-decisions.anth.us/models/