thb for me it's been the workflow around it. Things like:
- Pulling up my phone, dictating a quick thing and sending it of to work on one of my servers.
- Shooting things off to claude code, codex, grok, so fast. So well integrated.
- Hitting snooze on a session for next week
- hiding sessions in snoozed and working, so I can focus on what needs me now.
- Dragging filed straight into the chat
- Quickly looking at the diff on the side and go through files with the "viewed" button as in github.
- ctrl+j to get a terminal quickly to do something fast
- ... So many more but this is top of mind.
I just get so much more done, faster, with less context fatigue than before because of how it's laid out.
I read through the skill file of graphify and ran some of the commands it was telling the agent to run. The vibe I got is there was a lot of structured information returned that the model (and harness) isn't set up to work with effectively.
It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.
The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.
I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
Not exactly imo, you could spend a crapton of money with white box pen testing with a bunch of frontier llms, and it might become a bunch harder to break into (it generally costs more to attack than it does to defend). Not to mention formal verification might become possible for more devices which might make it nearly impossible to find exploits.
Imo, formal methods like more expressive/stricter type systems are key to making LLM generated code successful. Of course models will get better, but trusting the output will become much easier with a type system that proves more properties.
Inefficient per watt, yes, but local inference capacity is greatly underutilized in aggregate. If a model can run on a machine that already exists, that's a bunch of additional chips that don't need to be built.
You picked up on what I didn't write existing machines don't have to be built, you just use them.
It occurred to me much later that local machines could become a shared resource of a small co-op. Or it only runs off solar or energy harvested at times of excess generation (too much daytime solar, lots of wind, or overnight when demand naturally drops).
Exploring how to partition inference across many machines and shed load for idle sections of hardware would be a problem that could be solved by a community of smart people like us here