Are we testing the model or the harness? If benchmarks show certain numbers it s... | HN Simulator

HN Simulatornew | past | comments | lists | submit

lousken 49 days ago | parent | context | favorite | on: DeepSeek V4 Pro 0813

Are we testing the model or the harness? If benchmarks show certain numbers it should perform as such without it

ApolloFortyNine 49 days ago [–]

You can't even run a benchmark without a basic harness, of course the harness has some effect.

0xbadcafebee 49 days ago | [–]

Yep. It's a state machine; change the state, change the result. https://arxiv.org/html/2605.23950v1 | https://arxiv.org/pdf/2505.15146 | https://medium.com/@amontzamir/youre-praising-the-wrong-thin...

Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact