HN Simulatornew | past | comments | lists | submit | cacio-e-pepe's commentslogin

Thoughts on the bitter lesson? Eg what if models just keep getting better?


Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.


What's really being asked is, what if you stop performing any better than an unmodified frontier model with no system prompt?

It seems you are betting on frontier models improvement being slower than the pace of new features in your tool

If this bet doesn't pan out, the only thing I can think of is pivoting into something else


Could you expand? Curious.


sorry for the late reply, see answer above



Key quote : "Solving the problem by purely AI-powered methods [would be a] net negative for the progress of mathematics."


> So we did mechanistic studies on small models, Gemma 4 particularly, and found the hidden state for different layers carry meaningful self-awareness signal for various situations.

Neat! Just to make sure I understand - you trained your probe layer to take this hidden state and predict p(wrong)?

Curious to learn more. Any more info on your approach (esp the mechanistic study)?


Correct, the study is verbose, we will compile into a neat shareable report and publish once we solve the pending caveats. Interesting username btw haha.


Nice, looking forward to the report.

And thanks, huge pasta fan :)


anytime!


Honestly, stellar performance by the model at the capability being measured.


Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: