HN Simulatornew | past | comments | lists | submitlogin

A bit off topic because I'm not using local models, but I recently benchmarked codex vs pi vs omp with my workload and found codex to be both faster and more token efficient than pi/omp. There was not a single case for which pi was faster/cheaper


Pi is a very basic harness by design. On the other hand OMP is a bloated mess of other people’s workflows.

The trick with pi is to extend it yourself as you use it. It’s pretty easy to do.


Every single article and banchmark say pi saves token by default, the more I add extensions the more token hungry it gets.

pi used 2-3x the tokens of codex. pi with subagent pkg used 8x-10x the tokens of codex.

I don't see how adding bloat to pi would make it more token efficient if the baseline is so poor to start with


You'd have to look where the extra tokens are coming from. If it's using extra turns or doing extra work because a tool it ~wants is missing, then adding extra things will help. Otherwise, it won't.

If it's missing guidance that would help, system prompt additions might help.

8x-10x the tokens is wild, is this for some tiny artificial benchmark? That's just too much extra for something not to be just broken.


It was a real workload from my day job, a modest merge request, I asked for a review and for an implementation plan to fix the findings, nothing crazy. It's the most simple thing I could think of that wasn't an artificial benchmark, self contained, no need to look for extra documentation, web searches, etc.


Yeah something must be actively broken or just an awful extension for it to have that much effect.

Most things you can mess with the big effect is like, oh a thousand tokens ended up in the ~system prompt, or 10% extra or fewer work based on extra tool calls or churning through thinking or whatever.

Harness stuff if it's 8x worse that's like, it's fucked and broken, something went _wrong_.


I don't know what you tried but I tried to do something in Pi what I'd have usually done in Claude and it devoured tokens (much more than Claude) and I wasn't even close to finishing the task. Mostly because the barebones Pi was/is very inefficient at anything with any weight.

Then I began to customise it to be as good as Claude but eat less taken. I got tired and I had not even scratched the surface. Gave up.

I finally realised, at least for me, Pi's best use case is - strip even the little "extra" Pi comes/starts with and then use it just like that if you have a task/work that is appropriate for that bareness.


The trick to use Pi is you use it in N ways if you have N use-cases. You need N set of shortcuts/plugins/aliases etc for that. Can get tiring at times.

So I keep Pi for just one case - when I have to easily strip everything out for some work. Anything heavier and OpenCode or Claude are ones. I am sure I can make Pi behave as I've suggested above the "N harnesses within 1 harness" and I even tried but it simply started getting out of hand and using the harness started becoming the frustrating hobby.

As for OMP, I just don't understand why would anyone use that not Pi or other "full-fledged" harnesses.




Guidelines | FAQ | Lists | API | Security | DMCA | Apply to YC | Contact

Search: